Shell Script to Read a Json File Linux

Fatmawati Achmad Zaenuri/Shutterstock

JSON is one of the about popular formats for transferring text-based information around the web. It's everywhere, and you lot're bound to come beyond it. We'll show you how to handle information technology from the Linux command line using the jq command.

JSON and jq

JSON stands for JavaScript Object Notation. Information technology's a scheme that allows information to be encoded into plain text files, in a self-describing way. There are no comments in a JSON file—the contents should be self-explanatory. Each information value has a text string chosen a "proper name" or "key." This tells you what the data value is. Together, they're known as name:value pairs, or fundamental:value pairs. A colon (:) separates a key from its value.

An "object" is a collection of key:value pairs. In a JSON file, an object begins with an open curly brace ({) and ends with a endmost brace (}). JSON also supports "arrays," which are ordered lists of values. An array begins with an opening bracket ([) and ends with a endmost one (]).

From these elementary definitions, of course, arbitrary complexity tin arise. For example, objects tin be nested inside objects. Objects tin comprise arrays, and arrays can also contain objects. All of which can have open-ended levels of nesting.

In practice, though, if the layout of JSON data is convoluted, the design of the data layout should probably utilize a rethink. Of course, if you're not generating the JSON data, merely trying to use it, you have no say in its layout. In those cases, unfortunately, you just have to deal with it.

About programming languages have libraries or modules that allow them to parse JSON data. Sadly, the Bash shell has no such functionality.

Necessity being the mother of invention, though, the jq utility was built-in! With jq, we can easily parse JSON in the Bash beat out, or even convert XML to JSON. And information technology doesn't thing whether you have to work with well-engineered, elegant JSON, or the stuff nightmares are made of.

How to Install jq

Nosotros had to install jq on all the Linux distributions we used to enquiry this article.

To install jq on Ubuntu type this control:

sudo apt-go install jq

To install jq on Fedora, type this command:

sudo dnf install jq

To install jq on Manjaro, type this command:

sudo pacman -Sy jq

How to Make JSON Readable

JSON doesn't care virtually white space, and layout doesn't affect it. As long as it follows the rules of JSON grammer, systems that process JSON can read and understood it. Considering of this, JSON is often transmitted every bit a elementary, long string, without any consideration of layout. This saves a bit of space because tabs, spaces, and new-line characters don't accept to be included in the JSON. Of grade, the downside to all this is when a human being tries to read it.

Let's pull a curt JSON object from the NASA site that tells us the position of the International Space Station. We'll use roll, which tin can download files to retrieve the JSON object for us.

We don't care most whatsoever of the status messageswhorl usually generates, and so we'll blazon the following, using the -s (silent) option:

scroll -s http://api.open-notify.org/iss-now.json

Now, with a bit of effort, y'all tin read this. You have to pick out the information values, but information technology isn't easy or user-friendly. Allow's repeat this, but this time we'll pipe information technology through jq.

jq uses filters to parse JSON, and the simplest of these filters is a period (.), which means "impress the entire object." Past default, jq pretty-prints the output.

We put information technology all together and blazon the post-obit:

curl -south http://api.open-notify.org/iss-now.json | jq .

That's much better! Now, we tin encounter exactly what's going on.

The entire object is wrapped in curly braces. It contains two key:name pairs: message and timestamp. It also contains an object called iss_position, which contains two key:value pairs:longitude and latitude.

Nosotros'll try this once more. This time we'll blazon the following, and redirect the output into a file called "iss.json":

curl -s http://api.open-notify.org/iss-now.json | jq . > iss.json

true cat iss.json

This gives the states a well laid out re-create of the JSON object on our hard drive.

RELATED: How to Use curl to Download Files From the Linux Command Line

Accessing Data Values

Every bit we saw to a higher place,jq tin extract data values existence piped through from JSON. Information technology can too work with JSON stored in a file. Nosotros're going to work with local files so the command line isn't chaotic with curl commands. This should brand information technology a scrap easier to follow.

The simplest way to extract data from a JSON file is to provide a key proper noun to obtain its data value. Type a period and the key name without a infinite betwixt them. This creates a filter from the key proper name. Nosotros also need to tell jq which JSON file to employ.

We type the following to remember the bulletin value:

jq .message iss.json

jq prints the text of the message value in the terminal window.

If you have a fundamental name that includes spaces or punctuation, you have to wrap its filter in quotation marks. Care is usually taken to apply characters, numbers, and underscores just and so the JSON key names are not problematic.

First, nosotros blazon the following to retrieve the timestamp value:

jq .timestamp iss.json

The timestamp value is retrieved and printed in the last window.

Only how can we access the values inside theiss_position object? We tin use the JSON dot notation. We'll include the iss_position object name in the "path" to the primal value. To practise this, the name of the object the key is within will precede the name of the key itself.

Nosotros type the following, including the latitude key name (notation there are no spaces between ".iss_position" and ".breadth"):

jq .iss_position.breadth iss.json

To extract multiple values, you have to practise the post-obit:

• List the fundamental names on the command line.
• Separate them with commas (,).
• Enclose them in quotation marks (") or apostrophes (').

With that in mind, we type the post-obit:

jq ".iss_position.latitude, .timestamp" iss.json

The two values impress to the concluding window.

Working with Arrays

Allow's grab a different JSON object from NASA.

This time, we'll use a list of the astronauts who are in space correct now:

curl -southward http://api.open up-notify.org/astros.json

Okay, that worked, so permit's practise it once again.

We'll type the following to pipe it through jq and redirect information technology to a file chosen "astro.json":

curl -s http://api.open-notify.org/astros.json | jq . > astro.json

At present let's type the following to check our file:

less astro.json

As shown beneath, we at present see the listing of astronauts in infinite, likewise as their spacecrafts.

This JSON object contains an array called people. Nosotros know it'southward an array because of the opening bracket ([) (highlighted in the screenshot above). It'south an array of objects that each incorporate 2 key:value pairs:proper noun and craft.

Like we did before, we tin can use the JSON dot notation to access the values. We must also include the brackets ([]) in the proper name of the assortment.

With all that in mind, we blazon the following:

jq ".people[].name" astro.json

This fourth dimension, all the name values impress to the terminal window. What nosotros asked jq to do was print the name value for every object in the assortment. Pretty cracking, huh?

We tin retrieve the name of a single object if we put its position in the array in the brackets ([]) on the command line. The array uses naught-offset indexing, pregnant the object in the first position of the array is goose egg.

To admission the last object in the array you lot can use -1; to get the second to terminal object in the array, you can apply -2, and so on.

Sometimes, the JSON object provides the number of elements in the array, which is the instance with this one. Along with the assortment, information technology contains a cardinal:proper noun pair chosen number with a value of six.

The following number of objects are in this array:

jq ".people[1].proper noun" astro.json

jq ".people[iii].name" astro.json

jq ".people[-1].name" astro.json

jq ".people[-2].name" astro.json

You tin can too provide a offset and stop object within the array. This is called "slicing," and it tin can be a piffling disruptive. Remember the array uses a zero-offset.

To retrieve the objects from index position two, up to (simply not including) the object at index position iv, we type the following command:

jq ".people[2:four]" astro.json

This prints the objects at array alphabetize two (the third object in the assortment) and iii (the fourth object in the array). It stops processing at array alphabetize 4, which is the 5th object in the array.

The manner to better empathize this is to experiment on the control line. You'll shortly see how information technology works.

How to Use Pipes with Filters

You can pipe the output from ane filter to another, and you lot don't have to learn a new symbol. The same as the Linux command line,jq uses the vertical bar (|) to stand for a pipe.

We'll telljq to pipe the people assortment into the .name filter, which should list the names of the astronauts in the terminal window.

Nosotros blazon the following:

jq ".people[] | .proper noun" astro.json

RELATED: How to Use Pipes on Linux

Creating Arrays and Modifying Results

We can use jq to create new objects, such every bit arrays. In this case, we'll extract iii values and create a new array that contains those values. Note the opening ([) and closing brackets (]) are also the first and last characters in the filter cord.

Nosotros type the following:

jq "[.iss-position.latitude, iss_position.longitude, .timestamp]" iss.json

The output is wrapped in brackets and separated by commas, making information technology a correctly formed array.

Numeric values can also be manipulated as they're retrieved. Allow'due south pull the timestamp from the ISS position file, and and so extract information technology once more and change the value that'southward returned.

To practise and so, nosotros type the post-obit:

jq ".timestamp" iss.json

jq ".timestamp - 1570000000" iss.json

This is useful if yous demand to add or remove a standard offset from an array of values.

Allow's type the following to remind ourselves what the iss.json file contains:

jq . iss.json

Let's say we want to get rid of the message key:value pair. It doesn't accept anything to exercise with the position of the International Infinite Station. It'due south merely a flag that indicates the location was retrieved successfully. If it'southward surplus to requirements, we can dispense with it. (Yous could besides just ignore it.)

Nosotros tin can use jq's delete function,del(), to delete a key:value pair. To delete the message key:value pair, nosotros blazon this command:

jq "del(.message)" iss.json

Annotation this doesn't really delete information technology from the "iss.json" file; it merely removes information technology from the output of the command. If yous need to create a new file without the bulletin key:value pair in it, run the command, so redirect the output into a new file.

More than Complicated JSON Objects

Let's recollect some more NASA data. This time, we'll utilize a JSON object that contains information on meteor impact sites from around the world. This is a bigger file with a far more complicated JSON structure than those we've dealt with previously.

First, nosotros'll type the following to redirect information technology to a file chosen "strikes.json":

curl -south https://data.nasa.gov/resource/y77d-th95.json | jq . > strikes.json

To run into what JSON looks similar, we type the following:

less strikes.json

As shown below, the file begins with an opening bracket ([), so the entire object is an array. The objects in the array are collections of central:value pairs, and there's a nested object called geolocation. The geolocation object contains further cardinal:value pairs, and an array called coordinates.

Let's think the names of the meteor strikes from the object at index position 995 through the finish of the array.

We'll type the following to piping the JSON through three filters:

jq ".[995:] | .[] | .name" strikes.json

The filters function in the following ways:

• .[995:] : This tells jq to procedure the objects from array index 995 through the stop of the array. No number later on the colon ( : ) is what tellsjq to continue to the end of the array.
• .[] : This assortment iterator tells jq to procedure each object in the array.
• .name : This filter extracts the proper name value.

With a slight modify, we can excerpt the concluding ten objects from the array. A "-10" instructs jq to start processing objects 10 back from the end of the array.

Nosotros type the following:

jq ".[-10:] | .[] | .proper noun" strikes.json

Just as we have in previous examples, we can type the following to select a unmarried object:

jq ".[650].name" strikes.json

We tin also apply slicing to strings. To do so, nosotros'll type the following to request the first four characters of the name of the object at array index 234:

jq ".[234].name[0:4]" strikes.json

We can also run across a specific object in its entirety. To exercise this, we type the following and include an assortment index without any key:value filters:

jq ".[234]" strikes.json

If you desire to encounter only the values, y'all can do the same thing without the key names.

For our example, we type this control:

jq ".[234][]" strikes.json

To call back multiple values from each object, we separate them with commas in the following command:

jq ".[450:455] | .[] | .proper noun, .mass" strikes.json

If you want to call back nested values, you have to identify the objects that form the "path" to them.

For example, to reference the coordinates values, we have to include the extensive array, the geolocation nested object, and the nested coordinates assortment, as shown below.

To come across the coordinates values for the object at index position 121 of the array, we type the following command:

jq ".[121].geolocation.coordinates[]" strikes.json

The length Role

The jq length function gives unlike metrics according to what it's been applied, such as:

• Strings: The length of the string in bytes.
• Objects: The number of key:value pairs in the object.
• Arrays: The number of array elements in the array.

The following command returns the length of the proper name value in x of the objects in the JSON array, starting at alphabetize position 100:

jq ".[100:110] | .[].proper name | length" strikes.json

To see how many fundamental:value pairs are in the first object in the array, we blazon this command:

jq ".[0] | length" strikes.json

The keys Function

You tin use the keys function to find out about the JSON you lot've got to piece of work with. It can tell you what the names of the keys are, and how many objects at that place are in an array.

To observe the keys in the people object in the "astro.json" file, nosotros blazon this command:

jq ".people.[0] | keys" astro.json

To encounter how many elements are in the people array, we blazon this control:

jq ".people | keys" astro.json

This shows in that location are half dozen, zero-offset array elements, numbered cipher to five.

The has() Part

Yous can employ the has() office to interrogate the JSON and see whether an object has a particular primal name. Notation the key name must exist wrapped in quotation marks. Nosotros'll wrap the filter command in single quotes ('), as follows:

jq '.[] | has("nametype")' strikes.json

Each object in the array is checked, as shown beneath.

If you lot want to check a specific object, yous include its index position in the array filter, as follows:

jq '.[678] | has("nametype")' strikes.json

Don't Get About JSON Without Information technology

The jq utility is the perfect example of the professional, powerful, fast software that makes living in the Linux world such a pleasure.

This was just a brief introduction to the common functions of this command—there'due south a whole lot more to it. Be certain to check out the comprehensive jq manual if you want to dig deeper.

RELATED: How to Convert XML to JSON on the Command Line

mathisnerld1977.blogspot.com

Source: https://www.howtogeek.com/529219/how-to-parse-json-files-on-the-linux-command-line-with-jq/

Share this post