Our Blog

Ongoing observations by End Point people

Making sense of XML/JSON items in the shell

Muhammad Najmi bin Ahmad Zabidi

By Muhammad Najmi bin Ahmad Zabidi
December 31, 2019

a shell

Working as a system administrator means I have to spend quite some time during my work (and even during casual surfing) with the terminal. Sometimes I feel that certain information I want could just be fetched and parsed through the terminal, without having to use my mouse and point to the browser.

Some of the websites I visit use XML and JSON, which we could parse with Bash scripting. Previously I wrote a Ruby script to call Nokogiri to parse the XML elements until I found a Bash tool that could do the same thing.

These tools have already been around for quite a while—I’d just like to share what I did with them. The tools I used are xmlstarlet for XML parsing and jq for JSON.


I have the following XML elements, and I’ll save them to a file called data.xml:

<rss version="2.0"

        <title>eSolat JAKIM : Waktu Solat Hari Ini</title>
        <link>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam</link>
        <description>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam</description>
        <dc:rights>Copyright JAKIM</dc:rights>
        <dc:date>26-12-2019 00:37:31</dc:date>
        <admin:generatorAgent rdf:resource="expressionengine" />


I’ll use xmlstarlet, together with a bunch of the related flags to parse these elements into something which is more eye-friendly.

xmlstarlet sel -T -t -n \
  -o "------------------------------" -n \
  -o "Area: " \
  -m "//channel" -v "link" -n  \
  -o "Date: " \
  -m "//channel" -v "dc:date" -n  \
  -o "------------------------------" -n \
  -t -m "//item" -o "Time: " -v "title" \
  -o " -- " \
  -o "Time: " -v "description" -n  \

The output looks like this:

Area: Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam
Date: 26-12-2019 00:37:31
Time: Imsak -- Time: 05:53:00
Time: Subuh -- Time: 06:03:00
Time: Syuruk -- Time: 07:14:00
Time: Zohor -- Time: 13:16:00
Time: Asar -- Time: 16:39:00
Time: Maghrib -- Time: 19:13:00
Time: Isyak -- Time: 20:28:00

I’ll put this in a Bash script, and call it xmlstarlet-time.sh.



if [[ -z $1 ]]; then
  echo "Put the location code"
  echo "$0 <location code>"
  echo -n

lynx -source "https://www.e-solat.gov.my/index.php?r=esolatApi/xmlfeed&zon=$1" > $XMLPATH

xmlstarlet sel -T -t -n \
  -o "------------------------------" -n \
  -o "Area: " -m "//channel" -v "link" -n  \
  -o "Date: " -m "//channel" -v "dc:date" -n  \
  -o "------------------------------" -n \
  -t -m "//item" -o "Time: " -v "title" -o " -- " -o "Time: " -v "description" -n  \

Now, after making it executable with chmod +x xmlstarlet-time.sh, I can just run the script whenever I need the info. In my case, I would type ./xmlstarlet-time.sh SGR01 in order to get the above information. I got the code (in my case) from the XML URL above. Your use case will likely be different.


Let’s say I want to grab the latest currency exchange, taking the base of USD from exchangeratesapi.io. I can use curl to get the data.

$ curl -s 'https://api.exchangeratesapi.io/api/latest?base=USD'

Which will return:


Using jq, we can format the information more readably:

$ curl -s 'https://api.exchangeratesapi.io/api/latest?base=USD' | jq
  "rates": {
    "CAD": 1.3160649819,
    "HKD": 7.7879061372,
    "ISK": 122.3826714801,
    "PHP": 50.8402527076,
    "DKK": 6.7429602888,
    "HUF": 299.4223826715,
    "CZK": 23.0009025271,
    "GBP": 0.7719584838,
    "RON": 4.3131768953,
    "SEK": 9.4361913357,
    "IDR": 13985.0180505415,
    "INR": 71.2567689531,
    "BRL": 4.0835740072,
    "RUB": 62.0877256318,
    "HRK": 6.719765343,
    "JPY": 109.3772563177,
    "THB": 30.155234657,
    "CHF": 0.9817689531,
    "EUR": 0.9025270758,
    "MYR": 4.1364620939,
    "BGN": 1.7651624549,
    "TRY": 5.9561371841,
    "CNY": 7.0074909747,
    "NOK": 8.94566787,
    "NZD": 1.5086642599,
    "ZAR": 14.1935018051,
    "USD": 1,
    "MXN": 18.9626353791,
    "SGD": 1.3553249097,
    "AUD": 1.4457581227,
    "ILS": 3.4714801444,
    "KRW": 1162.1931407942,
    "PLN": 3.8445848375
  "base": "USD",
  "date": "2019-12-24"

Next, I can make use of the tool in my shell script.

#!/bin/bash -l


if [ -z `which jq` ]; then
  printf "You need to install jq, a JSON parsing tool \n"

sourcemoney=$(echo $2 | tr '[:lower:]' '[:upper:]')
target=$(echo $3 | tr '[:lower:]' '[:upper:]')

sumber=$(curl -s "https://api.exchangeratesapi.io/api/latest?base=$sourcemoney" | jq . | grep -i $target | awk -F ':|,' '{ print $2 }')

jumlah=$(printf "%f*%f\n" $1 $sumber | bc)

printf "Price per unit: ${GREEN}1 $sourcemoney${NC} = ${YELLOW}$target %.2f${NC}\n" $sumber

echo -e "Source money: ${YELLOW}$sourcemoney $1${NC}"
echo -n

printf "Total money after the conversion: ${YELLOW}$target %.2f ${NC}\n" $jumlah

Then I can save the script into a file called moneychanger-with-api.sh and make it executable with chmod +x moneychanger-with-api.sh.

And now the script will do the parsing for me, without the need for a browser.

$ ./moneychanger-with-api.sh 100 usd eur
Price per unit: 1 USD =  EUR 0.90
Source money: USD 100
Total money after the conversion: EUR 90.25

$ ./moneychanger-with-api.sh 100 eur sgd
Price per unit: 1 EUR =  SGD 1.50
Source money: EUR 100
Total money after the conversion: SGD 150.17

shell json


Popular Tags


Search our blog