on fly Variable substitution

This is quite old case I had been using for long, but never blogged as it has security risk if not handled properly use of eval.

Using wide range of hosts services on distributed cluster it can be helpful.

Say we have list of hosts running different services, and you need to connect for particular reason the

Say we alias the list of hosts running services on different clusters,

• e.g. cluster: 
• ALPHA, BETA

• services: 
• FALCON, OOZIE

so we have aliases, host and services

ALPHA_FALCON_HOST=alpha001.my.home.net
ALPHA_OOZIE_HOST= alpha002.my.home.net
BETA_FALCON_HOST=beta1001.my.home.net
BETA_OOZIE_HOST=beta402.my.home.net

RANGE_HOST=beta402.my.home.net

we have aliases, service with port to access the service correctly

OOZIE_URL=’http://$OOZIE_HOST:11000/oozie’FALCON_URL=’http://$FALCON_HOST:15000’ 

Now we want to access BETA cluster OOZIE_HOST, and set the URL accordingly
setting two set of variables on fly to access there service,

OOZIE_HOST=$BETA_OOZIE_HOSTOOZIE_URL=`echo "$(eval echo $OOZIE_URL )"`

e.g. might be bad if you have few services,
but its very helpful if we have large cluster farm, and have proper aliasing

Code snippets on gist

Hi..

I will be further putting my code on https://gist.github.com/sanjeevtripurari

Blog will have just details, and link for the code..

now blogging seems boring will do more coding stuff..

:)

quick file copy (without ftp/scp)

This is for quick copy only, where you don't want to setup any ftp or scp server
and have the machines in same network

## say we have two hosts: alpha & beta
##  need to copy: file-big.tgz
## linux util: nc

on host: alpha
$ cat file-big.tgz | nc -v -l 3334


now get the file on host:beta
$ nc  -v  alpha 3334   > file-from-alpha-file-big.tgz


voila, we have the file on host beta...

milliseconds to date

Most of the java apps use milliseconds, here is bash date for getting the date for the same


## Current system date

$ date

Tue Jun  2 08:07:57 UTC 2015

## Epoch system date

$ date +"%s"

1433232485

## Milliseconds for the date

$ date +"%s%3N"

1433232489392

## Last 3 digits are for millisecs, so put decimal point and you get the get

$ date -d @"1433232489.392"

Tue Jun  2 08:08:09 UTC 2015

## or strip of Last 3 digits

$ date -d @"1433232489"

Tue Jun  2 08:08:09 UTC 2015

script for reading file or stream

I often wanted to pass the file to a pipeline some command, and also if the file is parameter to command it should accept


Voila!!

here is 'stream.sh'

#!/bin/bash

file=${1:--}

cat $file 

Usage and output

$ ./stream.sh underSTANDING.txt 

If you understand, say "understand".

If you don't understand, say "don't understand".

But if you understand and say "don't understand".

How do I understand that you understand? Understand!

$ cat underSTANDING.txt | ./stream.sh 

If you understand, say "understand".

If you don't understand, say "don't understand".

But if you understand and say "don't understand".

How do I understand that you understand? Understand!

$ ./stream.sh < underSTANDING.txt 

If you understand, say "understand".

If you don't understand, say "don't understand".

But if you understand and say "don't understand".

How do I understand that you understand? Understand!

bytes to gb/tb conversion

Often had to go to websites to do the bytes to KB.. TB.. GB conversions
here is the quick script for handy use

#!/bin/bash

# Give numeric value, which is in bytes

# will show all possible conversions 

call_bc() {

n1=$1

n2=$2

echo "scale=4; $n1/($n2)" |bc

}

nm=$1

pw=$2

pn=""

k_ilo=1024;

m_ega=$k_ilo*$k_ilo;

g_iga=$m_ega*$k_ilo;

t_era=$g_iga*$k_ilo;

p_eta=$t_era*$k_ilo;

[ -z "$nm" ] && exit

echo  "B: "$nm

for i in K M G T P; do

case $i in

K)pn=$k_ilo;;

M) let pn=$m_ega;;

G) let pn=$g_iga;;

T) let pn=$t_era;;

P) let pn=$p_eta;;

esac

echo -n "$i: "; call_bc $nm $pn

done

Usage and output

$ ./convert.sh 74274576

B: 74274576

K: 72533.7656

M: 70.8337

G: .0691

T: 0

P: 0

 

Creating date range for next few hours

We often get cases when we need date ranges to be generated, so it can take care of hours, days, months properly.

Script: file dategen.sh

#!/bin/bash
_g_DT="$1"
_g_TM="$2"

_DT=`date -d "$_g_DT $_g_TM" +"%Y%m%d %H"`

echo "For: $_DT"

for i in `seq 0 5`; do
      _DT_gen=`date -d "$_DT +$i hours" +"%Y-%m-%dT%H:00"`
      echo -e "\t$_DT_gen"
done

E.g with Sep 30, at 23 hours

bash dategen.sh "2014-09-30" "23"


For: 20140930 23
 2014-09-30T23:00
 2014-10-01T00:00
 2014-10-01T01:00
 2014-10-01T02:00
 2014-10-01T03:00
 2014-10-01T04:00