Our Blog

Ongoing observations by End Point people

Making sense of XML/JSON items in the shell

By Muhammad Najmi bin Ahmad Zabidi
December 31, 2019

a shell

Working as a system administrator means I have to spend quite some time during my work (and even during casual surfing) with the terminal. Sometimes I feel that certain information I want could just be fetched and parsed through the terminal, without having to use my mouse and point to the browser.

Some of the websites I visit use XML and JSON, which we could parse with Bash scripting. Previously I wrote a Ruby script to call Nokogiri to parse the XML elements until I found a Bash tool that could do the same thing.

These tools have already been around for quite a while—I’d just like to share what I did with them. The tools I used are xmlstarlet for XML parsing and jq for JSON.

XML

I have the following XML elements, and I’ll save them to a file called data.xml:

<rss version="2.0"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
    xmlns:admin="http://webns.net/mvcb/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:content="http://purl.org/rss/1.0/modules/content/">

    <channel>
        <title>eSolat JAKIM : Waktu Solat Hari Ini</title>
        <link>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam</link>
        <description>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam</description>
        <dc:language>ms</dc:language>
        <dc:creator>www.e-solat.gov.my</dc:creator>
        <dc:rights>Copyright JAKIM</dc:rights>
        <dc:date>26-12-2019 00:37:31</dc:date>
        <admin:generatorAgent rdf:resource="expressionengine" />

        <item>
            <title>Imsak</title>
            <description>05:53:00</description>
        </item>
        <item>
            <title>Subuh</title>
            <description>06:03:00</description>
        </item>
        <item>
            <title>Syuruk</title>
            <description>07:14:00</description>
        </item>
        <item>
            <title>Zohor</title>
            <description>13:16:00</description>
        </item>
        <item>
            <title>Asar</title>
            <description>16:39:00</description>
        </item>
        <item>
            <title>Maghrib</title>
            <description>19:13:00</description>
        </item>
        <item>
            <title>Isyak</title>
            <description>20:28:00</description>
        </item>
    </channel>
</rss>

I’ll use xmlstarlet, together with a bunch of the related flags to parse these elements into something which is more eye-friendly.

xmlstarlet sel -T -t -n \
  -o "------------------------------" -n \
  -o "Area: " \
  -m "//channel" -v "link" -n  \
  -o "Date: " \
  -m "//channel" -v "dc:date" -n  \
  -o "------------------------------" -n \
  -t -m "//item" -o "Time: " -v "title" \
  -o " -- " \
  -o "Time: " -v "description" -n  \
  /path/to/data.xml

The output looks like this:

------------------------------
Area: Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam
Date: 26-12-2019 00:37:31
------------------------------
Time: Imsak -- Time: 05:53:00
Time: Subuh -- Time: 06:03:00
Time: Syuruk -- Time: 07:14:00
Time: Zohor -- Time: 13:16:00
Time: Asar -- Time: 16:39:00
Time: Maghrib -- Time: 19:13:00
Time: Isyak -- Time: 20:28:00

I’ll put this in a Bash script, and call it xmlstarlet-time.sh.

#!/bin/bash

XMLPATH=/home/seth/data.xml

if [[ -z $1 ]]; then
  echo "Put the location code"
  echo "$0 <location code>"
  echo -n
  exit
fi

lynx -source "https://www.e-solat.gov.my/index.php?r=esolatApi/xmlfeed&zon=$1" > $XMLPATH

xmlstarlet sel -T -t -n \
  -o "------------------------------" -n \
  -o "Area: " -m "//channel" -v "link" -n  \
  -o "Date: " -m "//channel" -v "dc:date" -n  \
  -o "------------------------------" -n \
  -t -m "//item" -o "Time: " -v "title" -o " -- " -o "Time: " -v "description" -n  \
  $XMLPATH

Now, after making it executable with chmod +x xmlstarlet-time.sh, I can just run the script whenever I need the info. In my case, I would type ./xmlstarlet-time.sh SGR01 in order to get the above information. I got the code (in my case) from the XML URL above. Your use case will likely be different.

JSON

Let’s say I want to grab the latest currency exchange, taking the base of USD from exchangeratesapi.io. I can use curl to get the data.

$ curl -s 'https://api.exchangeratesapi.io/api/latest?base=USD'

Which will return:

{"rates":{"CAD":1.3160649819,"HKD":7.7879061372,"ISK":122.3826714801,"PHP":50.8402527076,"DKK":6.7429602888,"HUF":299.4223826715,"CZK":23.0009025271,"GBP":0.7719584838,"RON":4.3131768953,"SEK":9.4361913357,"IDR":13985.0180505415,"INR":71.2567689531,"BRL":4.0835740072,"RUB":62.0877256318,"HRK":6.719765343,"JPY":109.3772563177,"THB":30.155234657,"CHF":0.9817689531,"EUR":0.9025270758,"MYR":4.1364620939,"BGN":1.7651624549,"TRY":5.9561371841,"CNY":7.0074909747,"NOK":8.94566787,"NZD":1.5086642599,"ZAR":14.1935018051,"USD":1.0,"MXN":18.9626353791,"SGD":1.3553249097,"AUD":1.4457581227,"ILS":3.4714801444,"KRW":1162.1931407942,"PLN":3.8445848375},"base":"USD","date":"2019-12-24"}

Using jq, we can format the information more readably:

$ curl -s 'https://api.exchangeratesapi.io/api/latest?base=USD' | jq
{
  "rates": {
    "CAD": 1.3160649819,
    "HKD": 7.7879061372,
    "ISK": 122.3826714801,
    "PHP": 50.8402527076,
    "DKK": 6.7429602888,
    "HUF": 299.4223826715,
    "CZK": 23.0009025271,
    "GBP": 0.7719584838,
    "RON": 4.3131768953,
    "SEK": 9.4361913357,
    "IDR": 13985.0180505415,
    "INR": 71.2567689531,
    "BRL": 4.0835740072,
    "RUB": 62.0877256318,
    "HRK": 6.719765343,
    "JPY": 109.3772563177,
    "THB": 30.155234657,
    "CHF": 0.9817689531,
    "EUR": 0.9025270758,
    "MYR": 4.1364620939,
    "BGN": 1.7651624549,
    "TRY": 5.9561371841,
    "CNY": 7.0074909747,
    "NOK": 8.94566787,
    "NZD": 1.5086642599,
    "ZAR": 14.1935018051,
    "USD": 1,
    "MXN": 18.9626353791,
    "SGD": 1.3553249097,
    "AUD": 1.4457581227,
    "ILS": 3.4714801444,
    "KRW": 1162.1931407942,
    "PLN": 3.8445848375
  },
  "base": "USD",
  "date": "2019-12-24"
}

Next, I can make use of the tool in my shell script.

#!/bin/bash -l

RED='\033[0;31m'
YELLOW='\033[1;33m'
GREEN='\033[0;32m'
NC='\033[0m'

if [ -z `which jq` ]; then
  printf "You need to install jq, a JSON parsing tool \n"
  exit
fi

sourcemoney=$(echo $2 | tr '[:lower:]' '[:upper:]')
target=$(echo $3 | tr '[:lower:]' '[:upper:]')

sumber=$(curl -s "https://api.exchangeratesapi.io/api/latest?base=$sourcemoney" | jq . | grep -i $target | awk -F ':|,' '{ print $2 }')

jumlah=$(printf "%f*%f\n" $1 $sumber | bc)

printf "Price per unit: ${GREEN}1 $sourcemoney${NC} = ${YELLOW}$target %.2f${NC}\n" $sumber

echo -e "Source money: ${YELLOW}$sourcemoney $1${NC}"
echo -n

printf "Total money after the conversion: ${YELLOW}$target %.2f ${NC}\n" $jumlah

Then I can save the script into a file called moneychanger-with-api.sh and make it executable with chmod +x moneychanger-with-api.sh.

And now the script will do the parsing for me, without the need for a browser.

$ ./moneychanger-with-api.sh 100 usd eur
Price per unit: 1 USD =  EUR 0.90
Source money: USD 100
Total money after the conversion: EUR 90.25

$ ./moneychanger-with-api.sh 100 eur sgd
Price per unit: 1 EUR =  SGD 1.50
Source money: EUR 100
Total money after the conversion: SGD 150.17

shell json


Comments

Popular Tags


Archive


Search our blog