How to Work with Custom Fields: Part Two

Recently some developers have been asking about vulnerability management (VM) custom fields attached to vulnerabilities. In attempting to answer their questions, it was discovered the API documentation was incorrect. Now the documentation has been fixed, I thought a blog on custom fields would be helpful.

If you’re curious about why this is Part Two, it is because both Part One and Part Two are about metadata. Part One is about Asset metadata while Part Two is about vulnerability metadata.

Custom fields are additional metadata to vulnerabilities. They add context. For example, when creating a CISA risk meter, the CISA custom field was created to indicate the vulnerability was on the CISA list. This blog will discuss how to search for and update custom fields with APIs. It will also discuss code that lists unique custom fields and their values.

Custom Field Search

To search for a custom field, the “Search Vulnerabilities” API is used. The query parameter custom_fields is used in the following manner:

custom_fields:[]=

For example:

https://api.kennasecurity.com/vulnerabilities/search?custom_fields:Category[]=good

If you want to search for more than one custom field, I recommend:

https://api.kennasecurity.com/vulnerabilities/search?custom_fields:Category[]=good&custom_fields:CISA[]=true

Note that custom_fields: with the custom field and [] after the custom field is required. Unfortunately, wildcarding is not allowed.  Some developers might think that this query parameter is poorly defined. I would agree, and we know about it, but we’re stuck with it for now.

Here are two code examples:

In build_cisa_risk_meter.py:

 80     # Assemble the search URL.
 81     page_size_query = f"per_page={SEARCH_PAGE_SIZE}"
 82     custom_field_query = f"custom_fields:{custom_field}[]=none"
 83     search_url = f"{base_url}vulnerabilities/search?{page_size_query}&{custom_field_query}"

And in search_custom_fields.py:

 38     page_size_query = f"per_page={page_size}"
 39     custom_field_query = f"custom_fields:{search_custom_field}[]={search_value}"
 40     search_url = f"{base_url}vulnerabilities/search?{page_size_query}&{custom_field_query}"

The script search_custom_fields.py is provided in the custom_fields directory of Kenna’s blog_samples repo as a simple custom field search script. It takes two command line parameters, custom_field name, and custom_field value, and returns the basic information on the vulnerabilities that contain the desired custom field and the custom field value. Feel free to enhance.

Custom Field Update

Let’s say you have a custom field, risk_accepted_expiration_date. You want a user interface or a small script to update the risk accepted expiration date. To do this you will need to use the “Update Vulnerabilities” API. It can be used to update custom fields of one vulnerability. Here is an example below from update_custom_field.py:

 16 def update_vuln(base_url, headers, vuln_id, custom_field_id, custom_field_value):
 17     update_url = f"{base_url}vulnerabilities/{vuln_id}"
 18     update_custom_field_id = f"{custom_field_id}"
 19     update_data = {
 20         "vulnerability": {
 21             "custom_fields": {
 22                 update_custom_field_id: custom_field_value
 23             }
 24         }
 25     }
 26 
 27     print(f"Update URL: {update_url}")
 28     print_json(update_data)
 29 
 30     # Invoke the update vulnerability endpoint.
 31     response = requests.put(update_url, headers=headers, data=json.dumps(update_data))
 32     if response.status_code != 204:
 33         print("Vulnerability Update API ", response, update_url)
 34         sys.exit(1)
 35 

Since this is an update, the custom field information is in the HTTP request body. To update a custom field, the custom field ID is required along with the new value to update with. To obtain the custom field ID, invoke the List or Search Vulnerabilities APIs.

Custom Field Bulk Update

To update a custom field for a list of vulnerabilities, the “Bulk Update Vulnerabilities” API is used. This was discussed in the “How to Create a CISA Risk Meter” blog, section “Bulk Vulnerabilities Update,” but I’m going to review it here. The HTTP request body is the same when it comes to custom fields;  however, now vulnerability_ids to be updated are included in the HTTP request body. Here is an example from build_cisa_risk_meter.py:

152 def update_cisa_vulns(base_url, headers, vuln_ids, custom_field_id):
153     if len(vuln_ids) == 0:
154         return
155 
156     bulk_update_url = f"{base_url}vulnerabilities/bulk"
157     update_custom_field_id = f"{custom_field_id}"
158     update_data = {
159         "vulnerability_ids": vuln_ids,
160         "vulnerability": {
161             "custom_fields": {
162                 update_custom_field_id: "true"
163             }
164         }
165     }
166 
167     response = requests.put(bulk_update_url, headers=headers, data=json.dumps(update_data))

Again the custom field ID along with the new custom field value is required.

List Unique Custom Fields

Saving the largest topic for last. One issue with using custom fields is there is no way to obtain a list of custom fields being used for all your vulnerabilities. Are they still valid? What values are you using? I wrote the script, blog_list_custom_fields.py, to create a list of custom fields with a usage count, and custom field values, and written to a CSV file.

Data Structures

Let’s look at the script’s data structures. There is a class, Custom_Field that contains the custom field name, ID, unique values, and count. It contains methods to add new custom field values, return the values, and return the number of values. Looking at it, you will notice that the code checks if the value is None.

 29     # Append a custom field value to the list of values.
 30     def append_new_value(self, custom_field_value):
 31         if custom_field_value is None:
 32             return
 33 
 34         self.custom_field_count += 1
 35 
 36         # If the value is not in the list of values, add it.
 37         if not custom_field_value in self.custom_field_values:
 38             self.custom_field_values.append(custom_field_value)

The reason is once a custom field is defined, it is attached to all vulnerabilities with a null value. This script is only interested in non-null values (None in python). If the value is not, it is set for this vulnerability and therefore placed in the values array provided that it is not already in the array.

The other data structure is the dictionary, unique_custom_fields. The custom field name is the key and it maps to an Custom_Field object.  It is passed by reference to the appropriate functions.

331     # A dictionary of custom fields keyed by custom field name with the value
332     # of a custom field object.
333     unique_custom_fields = {}

Obtaining Vulnerability Information

First, the script does a vulnerability export.  This is very similar to an asset export. This was covered in the blog “Unique Asset Tags – Part 1“.  The only difference is that model in export_settings is set to vulnerability instead of asset for the “Request Data Export” API.

    filter_params = {
        'status' : ['open'],
        'export_settings': {
            'format': 'jsonl',
            'model': 'vulnerability'
        }
    }

Please read “Unique Asset Tags – Part 1” to understand Data Exports. The vulnerability data export code produces a file vuln.jsonl. which contains all the vulnerabilities in a JSONL format.

process_vuln_export()

Reads the JSONL file line by line converting each line to a JSON dictionary. Each line is a vulnerability.  If there are custom_fields in the vulnerability, then further processing is required in process_custom_fields().  A dot is produced for every 1000 vulnerabilities processed.  Hopefully, this status won’t keep you hanging and indicates processing is begin performed.

241 # Process vulnerabilities in the JSONL format.
242 def process_vuln_export(jsonl_vuln_file_name, unique_custom_fields):
243     print_info(f"Opening {jsonl_vuln_file_name} for processing.")
244     logging_interval = 1000
245 
246     # Open the JSONL file and read it line by line, checking each vulnerability line for custom fields.
247     with open(jsonl_vuln_file_name, 'r') as jsonl_f:
248         for line_num, vuln_line in enumerate(jsonl_f):
249             vuln = convert_to_json(vuln_line)
250             if "custom_fields" in vuln:
251                 logging.info(f"Found custom_field in Vuln {line_num}")
252                 process_custom_fields(unique_custom_fields, vuln["custom_fields"])
253 
254             if (line_num + 1) % logging_interval == 0:
255                 print(".", end='', flush='True')
256 
257     print("")
258     return (line_num + 1)

process_custom_fields()

This function decides in the custom field is unique. It does this by checking if the custom field name key is in unique_custom_fields (line 232).

224 # Process an array of custom fields in a vulnerability.
225 def process_custom_fields(unique_custom_fields, custom_fields):
226 
227     # Process one custom field at a time.
228     for custom_field in custom_fields:
229         cf_name = custom_field["name"]
230 
231         logging.debug(f"Processing custom field: {cf_name}")
232         if cf_name in unique_custom_fields:
233             unique_custom_field = unique_custom_fields[cf_name]
234             if unique_custom_field.custom_field_id != custom_field["custom_field_definition_id"]:
235                 print_error(f"IDs for custom field {cf_name} do not match.  " +
236                             f"{unique_custom_field.custom_field_id}, {custom_field['custom_field_definition_id']}")
237             unique_custom_field.append_new_value(custom_field["value"])
238         else:
239             append_custom_field(unique_custom_fields, custom_field)
240 

If the custom field is in the dictionary, then the value is added to the custom field by calling append_new_value() in line 237, else a new custom field entry is created by calling append_custom_field() in line 239.

append_custom_field()

This is the function where a new Custom_Field object is created and appended to the unique_custom_fields dictionary.

218 # Append a custom field to the dictionary of unique custom fields.
219 def append_custom_field(unique_custom_fields, custom_field):
220     cf_name = custom_field["name"]
221     a_custom_field = Custom_Field(custom_field)
222     unique_custom_fields[cf_name] = a_custom_field

As a reminder, the custom field name is used as a key.

write_csv_file()

Finally, we get to the function that writes the unique custom field information to the CSV file, uniq_custom_fields.csv.

260 # Write the information out to a CSV file.
261 def write_csv_file(custom_fields):
262     # Open the CSV file and write the header row.
263     csv_file_name = "uniq_custom_fields.csv"
264     uniq_custom_fields_fp = open(csv_file_name, 'w', newline='')
265     uniq_custom_field_writer = csv.writer(uniq_custom_fields_fp)
266     uniq_custom_field_writer.writerow(["Custom Field", "Custom Field ID", "Field Count", "Value Count", "Custom Field Values"])
267 
268     # Process each custom field and write it to the CSV file.
269     for custom_field_key in custom_fields:
270         custom_field = custom_fields[custom_field_key]
271 
272         if custom_field.custom_field_count == 0:
273             uniq_custom_field_writer.writerow([custom_field.custom_field_name, custom_field.custom_field_id, custom_field.custom_field_count])
274         else:
275             uniq_custom_field_writer.writerow([custom_field.custom_field_name, custom_field.custom_field_id,
276                                                custom_field.custom_field_count, str(custom_field.get_num_values()), custom_field.get_values()])
277 
278     uniq_custom_fields_fp.close()
279     print_info(f"{csv_file_name} is now available.")
280 

There are two types of rows, one for null custom field values and one for non-null custom field values.

• null value row contains: custom field name, ID, and a zero count. (line 273)
• a non-null row contains: custom field name, ID, non-zero count, number of unique values, and the unique values. (line 275-276)

If the count is zero, that implies that no one is currently using the custom field.  It could be time to remove the unused custom field.

CSV Output

Above is an Excel representation of the unique_custom_fields.csv. As you can see the spreadsheet has “Custom Field,” “Custom Field ID,” “Field Count,” “Value Count,” and “Custom Field Values.” If the “Field Count” is zero, the custom field is currently not being used. “Custom Field Values is a comma separate string; for example, “Bad, good, extract, excellent” are contained in one cell.

Conclusion

After reading this blog, my hope is as a developer, you will know how to work with VM custom fields and the code examples will inspire you to write your own scripts. The new code mentioned in this blog is located in Kenna Security’s Github blog_samples repository under the python/custom_fields directory.

Until next time,

Rick Ehrhart

API Evangelist