Re: UNIX : script help/input

From: Don Yu <donyu_at_jhu.edu>
Date: Thu, 02 Oct 2003 06:34:29 -0800
Message-ID: <F001.005D1D0B.20031002063429@fatcity.com>

Johan:

Please see the attachment file, which is my script for everyday web server log file and uses nslookup to process the results. Hope that it is helpful.

Don

Johan Muller wrote:

> Anybody with a quick and dirty (elegant would be nice too), to munge
> output from a nslookup output file to a delimited file?
>
> 'File content:
>
> Server: dns1.mci.com
> Address: 199.249.19.1
>
> Name: WCOM-4NXZGAPWY5.mcilink.com
> Address: 166.50.73.209
>
> Delimited file should have the following line(s); (using | or
> whatever as delimiter):
>
> '166.50.73.209'|'4NXZGAPWY5.mcilink.com'.
>
> Required for both single and multiple records.
>
>
>
> TIA

#! /bin/ksh
# file name: log_file_parsing
# Date: March 6,2003; Mar 7,2003; Mar 10,2003; Mar 13,2003; May 7,2003; 12 May,2003; May 15,2003; 
#       May 20,2003; May 30,2003
#
# Author: Don Y.
# Desc: an example of parsing netTracker related web access from web server (www.welch.jhu.edu) access 
#       log file. Extract client IP, access date, and host IP. This script also transfer client IP
#       with alphabet  into digital one by issuing nslookup command call. Finally make nslookup work
#       for it outputs a few lines instead of one line.  
#

# define output file
WORKING_DIR=$1
OUTPUT_FILE="$WORKING_DIR/log_file_parsing_output.temp" INPUT_FILE=$WORKING_DIR/$2
TEMP_NAME=$3
SCRIPT_HOME="/export/users/dony/report"

# check input access log file exists or not under $WORKING_DIR if [ ! -f $WORKING_DIR/$2 ]
then

	echo "input access log file doesn't exist at working directory!"
	exit 1

fi

# check output file under working directory exist or not : $WORKING_DIR/log_file_parsing_output.temp if [ ! -f $WORKING_DIR/log_file_parsing_output.temp ] then

	touch $WORKING_DIR/log_file_parsing_output.temp
	chown dony:staff $WORKING_DIR/log_file_parsing_output.temp

fi

# check all temporary files exist or not if [ -f $WORKING_DIR/log_file_parsing_temp01.temp ] then

        rm $WORKING_DIR/log_file_parsing_temp01.temp else

	# create an empty file with owner status: dony:staff
	touch $WORKING_DIR/log_file_parsing_temp01.temp
	chown dony:staff $WORKING_DIR/log_file_parsing_temp01.temp

fi

if [ -f $WORKING_DIR/log_file_parsing_temp02.temp ] then

        rm $WORKING_DIR/log_file_parsing_temp02.temp else

	touch $WORKING_DIR/log_file_parsing_temp02.temp
	chown dony:staff $WORKING_DIR/log_file_parsing_temp02.temp

fi

# file parsing for NetTracker associated lines success=0
cat $INPUT_FILE | grep "/cgi-bin/ntlinktrack.cgi" | awk '{print $1,substr($4,2,11),$7}' | sed 's/\/cgi-bin\/ntlinktrack.cgi?//' > $WORKING_DIR/log_file_parsing_temp01.temp let success=success+$?
sed 's/http:\/\/www.welch.jhu.edu\/cgi-bin\/ntlinktrack.cgi?//' $WORKING_DIR/log_file_parsing_temp01.temp > $WORKING_DIR/log_file_parsing_temp02.temp let success=success+$?

# get start line number here
whole_parsing_start_line=`cat $OUTPUT_FILE | wc -l`

# change the date format
for arg_each_line in `cat $WORKING_DIR/log_file_parsing_temp02.temp | awk '{ print $1"==="$2"==="$3 }'` do

	one_line=`echo $arg_each_line | sed 's/===/ /g'`
	# debug one line below if enabled
	#echo $one_line >> $WORKING_DIR/log_file_parsing_temp03.temp
	client_ip=`echo $one_line | awk '{ print $1 }' | sed 's/ //g'`
	org_date=`echo $one_line | awk '{ print $2 }' | sed 's/ //g'`
	oracle_date=`$SCRIPT_HOME/ldate $org_date`
	host_ip=`echo $one_line | awk '{ print $3 }' | sed 's/ //g'`
	# transfer client_ip from alphabet to digital
	client_addr=`echo "$client_ip" | egrep -i -e [a-z] | sed 's/ //g'`
	# get the length of client_addr
	LENGTH=`expr "$client_addr" : '.*'`
	if [ $LENGTH -gt 8 ]
	then
	# add pattern of [1-9]....[1-9] for parsing since nslookup output might be changed 
		/usr/sbin/nslookup $client_addr 2> $WORLING_DIR/$TEMP_NAME | awk '{ print $2 }' | egrep -e [0-9]'.'[0-9]'.' > nslookup_result
		NSR_LINES=`cat nslookup_result | wc -l`
		if [ $NSR_LINES -eq 2 ];then
			client_digit=`cat nslookup_result 2> $WORLING_DIR/$TEMP_NAME | awk '( NR == 2 ) { print $0 }'`
		elif [ $NSR_LINES -eq 3 ];then
			client_digit=`cat nslookup_result 2> $WORLING_DIR/$TEMP_NAME | awk '( NR == 3 ) { print $0 }'`
		fi
		# remove the result file
		rm nslookup_result 2> $WORLING_DIR/$TEMP_NAME
		# check whether or not the nslookup command gets a result,which it should be larger than 8
		LENGTH=`expr "$client_digit" : '.*'`
		if [ $LENGTH -gt 8 ]
		then
			client_ip=$client_digit
		fi
	fi
	# output the result
	echo "$client_ip $oracle_date $host_ip" >> $OUTPUT_FILE
	let success=success+$?

done

# get whole NetTracker related lines
whole_NetTracker_lines=`cat $WORKING_DIR/log_file_parsing_temp01.temp | wc -l`

# get whole after parsing lines
whole_parsing_end_line=`cat $OUTPUT_FILE | wc -l`

whole_parsing_lines=`expr $whole_parsing_end_line - $whole_parsing_start_line`

# compare these two lines
if [[ $whole_NetTracker_lines -ne $whole_parsing_lines ]] then

        let success=success+1
fi

if [[ $success -eq 0 ]]
then

	echo "parsing file successfully!"
	rm $WORKING_DIR/log_file_parsing_temp01.temp
	rm $WORKING_DIR/log_file_parsing_temp02.temp
else
	echo "parsing file unsuccessfully!"
	exit $success      # return an error code 

fi

2

-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: Don Yu
  INET: donyu_at_jhu.edu

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services
---------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

Received on Thu Oct 02 2003 - 09:34:29 CDT