Masking Sensitive Data with Logback
Masking sensitive data in logback logs is done by partially or fully replacing the client-sensitive data or NPI (nonpublic personal information) with some arbitrary encoded text. For example, the SSN information can be replaced with all star characters or we can remove the complete SSN information from the logs.
Generally, we can mask sensitive data in two ways.
The first approach (not recommended) is creating a few utility functions that create masked string representation of domain objects having sensitive information.
Logger.info("Transaction completed with details : " + CommonUtils.mask(trasaction));This approach is problematic because masking calls are scattered over all the application code. In the future, we are asked to mask data only in the production and pre-production environments then we may be changing the code in multiple places.
Similarly, if we identified that we missed one domain object from the masking process, then we may need to change the code in many places and many log statements.
The second approach is separating the masking logic from application code and putting this in Logback configuration. Now, the change in the masking logic will be central to the configuration file and layout handler. Application classes will not participate in any kind of masking logic.
Any change in masking logic or scope must be handled by the Logback through layout handler classes and configuration files. This option can easily be managed and this should be the preferred way of data masking in logs.
2. How to Mask Data with Logback
The data masking in Logback is done in two steps:
- Define the masking patterns with the help of regular expressions in logback.xml configuration file.
- Define a custom Layout class that will read the masking patterns and apply those pattern regex on the log message.
2.1. Masking Patterns in Configuration File
This is a slightly difficult part where you will be writing the regex pattern for information to be masked. Writing regular expressions to cover all kinds of formatted outputs may not be so easy, but once it is done you will thank yourself later.
Following is such a configuration to log the mask data using the console appender (for demo) and it masks only the email and SSN fields.
((?!000|666)42-(?!00)8-(?!0000)6) (\w+@\w+\.\w+) %d [%thread] %-5level %logger - %msg%n Note that we can easily enable or disable the masking in a particular environment by using the if-else like condition of Janino library.
org.codehaus.janino janino 3.1.6 In the given example, we have enabled data masking in the production environment and disabled it in all other environments. The ENV is a system property that returns the environment name where the application is running.
((?!000|666)48-(?!00)2-(?!0000)2) (\w+@\w+\.\w+) %d [%thread] %-5level %logger - %msg%n %d [%thread] %-5level %logger - %msg%n The second part of the solution is to read the masking patterns from the configuration and apply them in the log messages. This is rather a simple approach and can be achieved with a custom pattern handler.
The given pattern handler created a single regular expression by combining all patterns from the configuration and using OR operator. This pattern is applied to all log messages that need to be processed by this pattern handler.
We can customize the logic implemented in this handler to meet our own requirements.
import java.util.ArrayList; import java.util.List; import java.util.regex.Matcher; import java.util.regex.Pattern; import java.util.stream.Collectors; import java.util.stream.IntStream; import ch.qos.logback.classic.PatternLayout; import ch.qos.logback.classic.spi.ILoggingEvent; public class DataMaskingPatternLayout extends PatternLayout < private Pattern aplpliedPattern; private ListmaskPatterns = new ArrayList<>(); public void addMaskPattern(String maskPattern) < maskPatterns.add(maskPattern); aplpliedPattern = Pattern.compile( maskPatterns.stream() .collect(Collectors.joining("|")), Pattern.MULTILINE); >@Override public String doLayout(ILoggingEvent event) < return maskMessage(super.doLayout(event)); >private String maskMessage(String message) < //When masking is disabled in a environment if (aplpliedPattern == null) < return message; >StringBuilder sb = new StringBuilder(message); Matcher matcher = aplpliedPattern.matcher(sb); while (matcher.find()) < IntStream.rangeClosed(1, matcher.groupCount()).forEach(group -> < if (matcher.group(group) != null) < IntStream.range(matcher.start(group), matcher.end(group)).forEach(i ->sb.setCharAt(i, '*')); > >); > return sb.toString(); > >Let us see the data masking in action. I will be executing the demo code in production and non-production mode, both.
In non-production mode, we are not setting the system property ENV so data masking will not happen.
Logger logger = LoggerFactory.getLogger(Main.class); Map customer = new HashMap(); customer.put("id", "12345"); customer.put("ssn", "856-45-6789"); customer.put("email", "admin@email.com"); logger.info("Customer found : <>", new JSONObject(customer));21:02:18.683 [main] INFO com.howtodoinjava.demo.slf4j.Main - Customer found :
When we run the application in production mode, we can see the masked output.
//Production mode ON System.setProperty("ENV", "prod"); Logger logger = LoggerFactory.getLogger(Main.class); Map customer = new HashMap(); customer.put("id", "12345"); customer.put("ssn", "856-45-6789"); customer.put("email", "admin@email.com"); logger.info("Customer found : <>", new JSONObject(customer));21:03:07.960 [main] INFO com.howtodoinjava.demo.slf4j.Main - Customer found :
In this Logback tutorial, we learned to create custom PatternLayout to mask the sensitive data from application logs. The data masking patterns are centrally controlled from the configuration file and that makes this technique so useful.
We can extend this feature to make environment specific masking by the use of conditional tags from Janino library that Logback supports implicitly.
Mask Passwords with Logback?
The logback version 0.9.27 introduced replacement capability. Replacements support regular expressions. For example, if the logged message was «userid=alice, pswd=’my secret'», and the output pattern was
you just modify the pattern to
Note that the above makes use of option quoting.
The previous log message would be output as «userid=alice, pswd=’xxx'»
For blazing performance, you could also mark the log statement as CONFIDENTIAL and instruct %replace to perform replacement only for log statements marked as CONFIDENTIAL. Example,
Marker confidential = MarkerFactory.getMarker("CONFIDENTIAL"); logger.info(confidential, "userid=<>, password='<>'", userid, password); Unfortunately, the current version of logback does not yet support conditional replacements (based on markers or otherwise). However, you could easily write your own replacement code by extending ReplacingCompositeConverter. Shout on the logback-user mailing list if you need further assistance.
Solution 2
I believe Masking is an aspect of your business, not the aspect of any technology or logging system. There are situations where the passwords, national identities etc should be masked while storing them in the DB as well due to legal reasons. You should be able to mask the xml before giving it to the logger.
One way to do it is to run the XML through XSLT that does that making and then give it to logger for logging.
If you doesn’t want to do this then LogBack has Filters support that is one of the option (not the right one though).
But understand that any generic out of the box solution you are trying to find at the logging infrastructure level is going to be suboptimal as every log message is going to be checked for masking.
How to mask JAVA Object confidential/personal information in logs while Printing
Here you will see all steps to mask confidential/ information like credit card, CVV, Exp date, SSN, password etc. So that it will print in mask form as ****** so that unauthorize use will not misuse of others information.
Main concept to mask an Java for personal information by using reflection API and string override method by extending mask class.
Below are steps to masking Java Object :
- Extends classes with MaskData class.
- Override toString() method of object class and implement in MaskData Class.
- Use JAVA reflection apis to get all fields objects and change SPI fields with *****.
- Use replace digits methods method to replace digits with *.
Below are classes AccountDetail, Address and CredeitCard where only confidential information need to mask for Credit Card.
package com.model; import com.mask.object.MaskData; public class AccountDetail extends MaskData < private String firstName; private String lastName; private AddressDetail address; private CreditCardDetail creditCardDetail; public AccountDetail(String firstName, String lastName, AddressDetail address, CreditCardDetail creditCardDetail) < super(); this.firstName = firstName; this.lastName = lastName; this.address = address; this.creditCardDetail = creditCardDetail; >public String getFirstName() < return firstName; >public void setFirstName(String firstName) < this.firstName = firstName; >public String getLastName() < return lastName; >public void setLastName(String lastName) < this.lastName = lastName; >public AddressDetail getAddress() < return address; >public void setAddress(AddressDetail address) < this.address = address; >public CreditCardDetail getCreditCardDetail() < return creditCardDetail; >public void setCreditCardDetail(CreditCardDetail creditCardDetail) < this.creditCardDetail = creditCardDetail; >>
Address Class not having any confidential information.
package com.model; import com.mask.object.MaskData; public class AddressDetail extends MaskData < private String addressLine1; private String city; private String state; private String pincode; private String country; public AddressDetail(String addressLine1, String city, String state, String pincode, String country) < super(); this.addressLine1 = addressLine1; this.city = city; this.state = state; this.pincode = pincode; this.country = country; >public String getAddressLine1() < return addressLine1; >public void setAddressLine1(String addressLine1) < this.addressLine1 = addressLine1; >public String getCity() < return city; >public void setCity(String city) < this.city = city; >public String getState() < return state; >public void setState(String state) < this.state = state; >public String getPincode() < return pincode; >public void setPincode(String pincode) < this.pincode = pincode; >public String getCountry() < return country; >public void setCountry(String country) < this.country = country; >>
CreditCardDetail class is having confidential fields cardNumber,cvv and expDate only these fields need to mask.
package com.model; import com.mask.object.MaskData; public class CreditCardDetail extends MaskData < private String cardNumber; private String cvv; private String expDate; public CreditCardDetail(String cardNumber, String cvv, String expDate) < super(); this.cardNumber = cardNumber; this.cvv = cvv; this.expDate = expDate; >public String getCardNumber() < return cardNumber; >public void setCardNumber(String cardNumber) < this.cardNumber = cardNumber; >public String getCvv() < return cvv; >public void setCvv(String cvv) < this.cvv = cvv; >public String getExpDate() < return expDate; >public void setExpDate(String expDate) < this.expDate = expDate; >>
This is main class where masking these confidential fields by using JAVA reflection apis and override toString() method of Object Class.
package com.mask.object; import java.lang.reflect.Field; import java.util.Arrays; import java.util.HashSet; import java.util.Set; import java.util.regex.Matcher; import java.util.regex.Pattern; public class MaskData implements Cloneable < Set fieldSet=new HashSet(); String[] spiData = < "cardNumber", "cvv", "expDate" >; public Object clone() < return this; >@Override public String toString() < StringBuffer buffer=new StringBuffer(); try < Object object=super.clone(); printObjectStream(buffer,object); >catch (CloneNotSupportedException ex) < ex.printStackTrace(); >return buffer.toString(); > private void printObjectStream(StringBuffer buffer, Object object) < try < buffer.append(object.getClass().getCanonicalName()).append("[\n"); Object value=null; for (Field field : object.getClass().getDeclaredFields()) < if(fieldSet.add(field.getName())) < field.setAccessible(true); //System.out.println(field.getName()); buffer.append(field.getName() + "="); value = field.get(object); if(value!=null) < if(field.getType().isArray() |field.getType().getCanonicalName().startsWith("com.model")) < printObjectStream(buffer,value); >else if (Arrays.asList(spiData).contains(field.getName())) < //field.set(object, replaceDigits((String) field.get(object))); buffer.append(replaceDigits((String) value) ); >else < buffer.append( value ); >> > buffer.append("\n"); > buffer.append("]"); > catch (IllegalAccessException ex) < ex.printStackTrace(); >> private String replaceDigits(String text) < StringBuffer buffer = new StringBuffer(text.length()); Pattern pattern = Pattern.compile("\\d"); Matcher matcher = pattern.matcher(text); while (matcher.find()) < matcher.appendReplacement(buffer, "X"); >return buffer.toString(); > > Below is Test program to above code .
package com.mask.object; import com.model.AccountDetail; import com.model.AddressDetail; import com.model.CreditCardDetail; public class MaskJavaObject < public static void main(String[] args) < AddressDetail addressDetail = new AddressDetail("Noida City Center", "Noida", "UP", "India", "20310"); CreditCardDetail creditCardDetail = new CreditCardDetail("1234567890123456", "123", "12/90"); AccountDetail accountDetail = new AccountDetail("Saurabh", "Gupta", addressDetail, creditCardDetail); System.out.println(accountDetail); >> Console Output
com.model.AccountDetail[ firstName=Saurabh lastName=Gupta address=com.model.AddressDetail[ addressLine1=Noida City Center city=Noida state=UP pincode=India country=20310 ] creditCardDetail=com.model.CreditCardDetail[ cardNumber=XXXXXXXXXXXXXXXX cvv=XXX expDate=XX/XX ] ]
Related Posts
Below are some more masking ways for different type of data like XML, JSON and printing objects before logging , sending to page or transferring over network.