Decrypt HTTPS Traffic Using Wireshark And Key File

Wireshark is a useful tool in troubleshooting. Wireshark can decrypt SSL traffic as long as you have the server private key. This can be extremely useful, if you have to debug HTTPS traffic and cannot use HTTP instead.

First we will capture a HTTPS traffic for our testing. Here our HTTPS server’s ip address is 192.168.x.x and the port is default 443. I prefer to use tcpdump for packet capture but you can do it using the Wireshark.

The below command will capture all the encrypted traffic to and from from our server.

$ sudo tcpdump -w /tmp/ssl.pcap  -ni eth0 -s0  host 192.168.x.x port 443

The captured data will go to the ssl.pcap file. Once you have the captured packets in the file open it in the Wireshark. Use the “Follow TCP Stream” options and you can see the encrypted data.

Next thing we need is the server’s private key. Once you have the key file to decrypt the traffic, just goto “Edit -> Preferences”. Now on the left side menu choose “Protocols -> SSL”. Fill “RSA Key list” field in the format <host>, <port>, <protocol>, <key_file>. ie We will specify the server’s IP address, the port on which the server listens and the path to the server’s private key. The file format needed for the server’s private key is PEM. In our example it is 192.168.x.x, 443, https, /path/to/keyfile.pem.

Now Apply the setting and return to main window.

Now if you click on each row you can see a “Decrypted SSL Data (size) “ tab on the bottom of “Packet Bytes” frame. This tab will be shown if there is any decrypted data available.

You can now use the “Follow SSL Stream” option to view the decrypted data stream.

Happy decrypting 😉

[Java-Tip] Non-Blocking Method To Download Files From Web

The URLConnection class contains many methods that let you communicate with the URL over the network. But the URLConnection doesn’t provide a callback mechanism to know the data read progress. Java’s support of interfaces provides a mechanism by which we can get the equivalent of callbacks. The trick is to define a simple interface that declares the method we wish to be invoked and to notify the data read progress of a URLConnection.

We define our Event as follows.

// FileDownloadEvent.java

public interface FileDownloadEvent
{
    // This is just a regular method so it can return something or
    // take arguments if you like.
    public void dataReadProgress (int done, int total, byte data[]);
    public void done(boolean error);
}

There is two methods, the dataReadProgress() and thedone() method. We invoke the dataReadProgress() each time we read a chunk of data to notify the data read progress. We use the done() method to inform the data read is over or an error has happened.

The class that will signal the event needs to expect objects that implement the dataReadProgress interface and then invoke the dataReadProgress() method as appropriate.
We will keep a counter for the downloaded data and fire the dataReadProgress event each time we read a chunk of data.

// FileDownload.java

import java.net.URL;
import java.net.URLConnection;
import java.io.InputStream;
import java.io.DataInputStream;
import java.io.BufferedInputStream;
import java.util.Arrays;
import java.lang.Thread;
import java.lang.Runnable;

public class FileDownload extends Thread implements Runnable
{
    private FileDownloadEvent ie;
    private InputStream is = null;
    private DataInputStream dis = null;
    private int dataReadSize = 4096;
    private String downloadURL = null;

    public FileDownload (FileDownloadEvent event)
    {
        // Save the event object for later use.
        ie = event;
    }

    public void request (String url)
    {
        this.downloadURL = url;
	this.start();
    }

    //...
    public void run ()
    {
        boolean error = false;
        try {
            URL url = new URL(this.downloadURL);
	    URLConnection fdCon = url.openConnection();

            int total = fdCon.getContentLength();

            is = url.openStream();  // throws an IOException
            dis = new DataInputStream(new BufferedInputStream(is));

	    byte[] data = new byte[dataReadSize];
	    int progress = 0, n;
            while ((n = dis.read(data)) > 0) {
	        progress += n;
                this.ie.dataReadProgress (progress, total, data);
                Arrays.fill(data, (byte)0);
            }
        } catch (Exception e)
        {
            error = true;
        }
        this.ie.done(error);
    }
    // ...
}

The code that wishes to receive the event notification must implement the FileDownloadEvent interface and just pass a reference to itself to the event notifier or do as in the below code.

// Download.java

public class Download
{
    public static void main(String args[])
    {
        // Create the event notifier and pass ourself to it.
        FileDownload req = new FileDownload (new FileDownloadEvent() {
            // Define the actual handler for the event.
            public void dataReadProgress (int done, int total, byte[] data)
            {
                System.out.println("Progress: " + ((float)done/(float)total) * 100 + "%");
                // Do something with data...
            }
            public void done (boolean error)
            {
		System.out.println("Download Completed.");
                // Do something...
            }
        });

        req.request("http://somedomain/path/to/file.gz");

	// Do something
    }
}

That’s all there is to it. I hope use this simple Java idiom will be useful to someone.

Playing With Python And CouchDB

This page moved to: http://segfault.in/2010/11/04/playing-with-python-and-couchdb/

How To Expand Usable Storage Space In Ubuntu

1. Using LVM

For partitions created on Logical Volume Manager (LVM) (Linux feature) at install time, they can be resized easily by concatenating extents onto them or truncating extents from them over multiple storage devices without major system reconfiguration.

Caution: Deployment of the current LVM system may degrade guarantee against filesystem corruption offered by journaled filesystems such as ext3fs unless their system performance is sacrificed by disabling write cache of hard disk.

Run a df from terminal.

$ df
Filesystem	1K-blocks	Used	Available	Use%	Mounted on
/dev/mapper/VolGroup00-LogVol00	7935392	6773500	752292	91%	/
/dev/sda5	497829	20904	451223	5%	/boot
tmpfs	1037084	0	1037084	0%	/dev/shm
/dev/mapper/VolGroup00-LogVol01	70877776	14988144	51045372	23%	/home

We have two partitions here, / partition is about 8 Gb and the /home partition is about 71 Gb. What we are trying to do is to expand the / partition to 10 Gb by taking free space from /home.

For /home you do:

# sudo umount /home
# sudo e2fsck -f /dev/VolGroup00/LogVol01
# resize2fs /dev/VolGroup00/LogVol01 69G
# lvreduce -L-2G /dev/VolGroup00/LogVol01
# mount /home

For / you do:

# lvextend -L+2G /dev/VolGroup00/LogVol00
# resize2fs /dev/VolGroup00/LogVol00

e2fsck and resize2fs belong to package e2fsprogs.

After resizing you will get

$ df
Filesystem	1K-blocks	Used	Available	Use%	Mounted on
/dev/mapper/VolGroup00-LogVol00	9299624	6779304	2043564	77%	/
/dev/sda5	497829	20904	451223	5%	/boot
tmpfs	1037084	0	1037084	0%	/dev/shm
/dev/mapper/VolGroup00-LogVol01	68877776	14999888	51033628	23%	/home

Read the lvm-howto for detailed infotmation.

2. Mounting another partition

If you have an empty partition (e.g., “/dev/sdx”), you can format it with mkfs.ext3(1) and mount(8) it to a directory where you need more space. (You need to copy original data contents.)

$ sudo mv work-dir old-dir
$ sudo mkfs.ext3 /dev/sdx
$ sudo mount -t ext3 /dev/sdx work-dir
$ sudo cp -a old-dir/* work-dir
$ sudo rm -rf old-dir

3. Using symlink

This might be the easiest way. If you have an empty directory (e.g., “/path/to/emp-dir”) in another partition with usable space, you can create a symlink to the directory with ln(8).

$ sudo mv work-dir old-dir
$ sudo mkdir -p /path/to/emp-dir
$ sudo ln -sf /path/to/emp-dir work-dir
$ sudo cp -a old-dir/* work-dir
$ sudo rm -rf old-dir

4. Using aufs

If you have usable space in another partition (e.g., “/path/to/”), you can create a directory in it and stack that on to a directory where you need space with aufs. With aufs you can unite several directories into a single virtual filesystem.

$ sudo mv work-dir old-dir
$ sudo mkdir work-dir
$ sudo mkdir -p /path/to/emp-dir
$ sudo mount -t aufs -o br:/path/to/emp-dir:old-dir none work-dir

FreeBSD net.inet.ip Sysctls Explained

This page moved to: http://segfault.in/2010/10/22/freebsd-net-inet-ip-sysctls-explained/

FFmpeg Tricks You Should Know About

FFmpeg is a complete, cross-platform solution to record, convert and stream audio and video. It includes libavcodec – the leading audio/video codec library. FFmpeg is free software and is licensed under the LGPL or GPL depending on your choice of configuration options.

FFmpeg supports most of the popular formats, we don’t need to worry a lot about that. Formats supported by FFmpeg include MPEG, MPEG-4 (Divx), ASF, AVI, Real Audio/Video and Quicktime. To see a list of all the codecs/formats supported by FFmpeg, run the following command:

ffmpeg -formats

1. X11 grabbing

FFmpeg can grab the X11 display.

ffmpeg -f x11grab -r 24 -s cif -i :0.0 /tmp/out.mpg

0.0 is display.screen number of your X11 server, same as the DISPLAY environment variable.

ffmpeg -f x11grab -r 24 -s cif -i :0.0+10,20 /tmp/out.mpg

0.0 is display.screen number of your X11 server, same as the DISPLAY environment variable. 10 is the x-offset and 20 the y-offset for the grabbing.

ffmpeg -f x11grab -r 25 -s 800x600 -i :0.0 /tmp/outputFile.mpg

2. Convert Pictures To Movie

First, rename your pictures to follow a numerical sequence. For example, img1.jpg, img2.jpg, img3.jpg,… Then you may run:

ffmpeg -f image2 -i img%d.jpg /tmp/a.mpg

Notice that `%d’ is replaced by the image number.

`img%03d.jpg' means the sequence `img001.jpg', `img002.jpg', etc…

If you have large number of pictures to rename, you can use the following command to ease the burden. The command, using the bourne shell syntax, symbolically links all files in the current directory that match *jpg to the `/tmp' directory in the sequence of `img001.jpg', `img002.jpg' and so on.

 x=1; for i in *jpg; do counter=$(printf %03d $x); ln "$i" /tmp/img"$counter".jpg; x=$(($x+1)); done

If you want to sequence them by oldest modified first, substitute $(ls -r -t *jpg) in place of *jpg.

Then run:

  ffmpeg -f image2 -i /tmp/img%03d.jpg /tmp/a.mpg

The same logic is used for any image format that ffmpeg reads.

3. Video Conversions

Quick and dirty convert to flv

ffmpeg -i inputfile.mp4 outputfile.flv

This converts any media ffmpeg handles to flash. It would actually convert anything to anything, it’s based on the file extension. It doesn’t do ANY quality control, sizing, etc, it just does what it thinks is best.

Convert .flv to .3gp

ffmpeg -i file.flv -r 15 -b 128k -s qcif -acodec amr_nb -ar 8000 -ac 1 -ab 13 -f 3gp -y out.3gp

Download YouTube videos as .flv and convert them to .3gp for your mobile phone.

Convert AVI to iPhone MP4

ffmpeg -i .avi -f mp4 -vcodec mpeg4 -b 250000 -s 480?320 -acodec aac -ar 24000 -ab 64 -ac 2 [destination].mp4

for 4:3 aspect:

ffmpeg -i source-xvid.avi -s 480x320 -aspect 4:3 -b 768k -ab 64k -ar 22050 -r 30000/1001 OUT.mp4

for 16:9:

ffmpeg -i source-xvid.avi -s 480x320 -aspect 16:9 -b 768k -ab 64k -ar 22050 -r 30000/1001 OUT.mp4

Create a video that is supported by youtube:

ffmpeg -i mymovie.mpg -ar 22050 -acodec libmp3lame -ab 32K -r 25 -s 320x240 -vcodec flv
mytarget.flv

Takes an mpeg video and coverts it to a youtube compatible flv file.
The -r 25 sets the frame rate for PAL, for NTSC use 29.97

4. Audio Conversion

Convert RM file to mp3

ffmpeg -i input.rm -acodec libmp3lame -ab 96k output.mp3

Adjust the bitrate (-ab) as necessary. If omitted FFmpeg will use a default of 64 kb/s.

Converting WMV to MP3 using FFMPEG

ffmpeg -i audio1.wmv audio1.mp3

This will convert audio1.wmv file to audio1.mp3
Converting WMV to FLV using FFMPEG

ffmpeg -i audio1.wmv audio1.flv

This will convert audio1.wmv file to audio1.flv, this will generate only audio content
Converting AMR to MP3 using FFMPEG

ffmpeg -i audio1.amr -ar 22050 audio1.mp3

This will convert audio1.amr file to audio1.mp3 having audio rate 22.05 Khz
Converting aac to mp3 using FFMPEG

ffmpeg -i audio1.aac -ar 22050 -ab 32 audio1.mp3

This will convert audio1.aac to audio1.mp3 having audio rate 22.05 Khz and Audio BitRate 32Khz
Converting aac to mp3 using FFMPEG with MetaData

ffmpeg -i audio1.aac -ar 22050 -ab 32 -map_meta_data audio1.mp3:audio1.aac audio1.mp3

This will convert audio1.aac to audio1.mp3 having audio rate 22.05 Khz and Audio BitRate 32Khz and will copy the meta data from .aac file to .mp3 file

5. Audio Extraction

ffmpeg -i video.avi -f mp3 audio.mp3

Dumping Audio stream from flv (using ffmpeg)

ffmpeg -i input.flv -f mp3 -vn -acodec copy ouput.mp3

6. Record Audio and Video from webcam

To record video run ffmpeg with arguments such as these:

ffmpeg -f video4linux2 -s 320x240 -i /dev/video0 out.mpg

To record both audio and video run ffmpeg with arguments such as these:

ffmpeg -f oss -i /dev/dsp -f video4linux2 -s 320x240 -i /dev/video0 out.mpg

7. Copy Only A Part Of Video

Cut out a piece of film from a file. Choose an arbitrary length and starting time.

ffmpeg -vcodec copy -acodec copy -i orginalfile -ss 00:01:30 -t 0:0:20 newfile

-vcodec, you choose what video codec the new file should be encoded with. Run ffmpeg -formats E to list all available video and audio encoders and file formats.

copy, you choose the video encoder that just copies the file.

-acodec, you choose what audio codec the new file should be encoded with.

copy, you choose the audio encoder that just copies the file.

-i originalfile, you provide the filename of the original file to ffmpeg

-ss 00:01:30, you choose the starting time on the original file in this case 1 min and 30 seconds into the film

-t 0:0:20, you choose the length of the new film

newfile, you choose the name of the file created.

8. Join Multiple Video Files

A few multimedia containers (MPEG-1, MPEG-2 PS, DV) allow to join video files by merely concatenating them.

Hence you may concatenate your multimedia files by first transcoding them to these privileged formats, then using the humble cat command (or the equally humble copy under Windows), and finally transcoding back to your format of choice.

mkfifo orig1.mpg
mkfifo orig2.mpg
ffmpeg -i input1.avi -sameq -y orig1.mpg
ffmpeg -i input2.avi -sameq -y orig2.mpg

Merge files

cat orig1.mpg orig2.mpg | ffmpeg -f mpeg -i - -vcodec copy -acodec copy merged.mpg

Merge and convert to avi

cat orig1.mpg orig2.mpg | ffmpeg -f mpeg -i - -sameq -vcodec mpeg4 -acodec libmp3lame merged.avi

Notice that you should either use -sameq or set a reasonably high bitrate for your intermediate and output files, if you want to preserve video quality.

Also notice that you may avoid the huge intermediate files by taking advantage of named pipes, should your platform support it:

9. Removing Synchronization Problems Between Audio and Video

ffmpeg -i source_audio.mp3 -itsoffset 00:00:10.2 -i source_video.m2v target_video.flv

This assumes that there is a 10.2 sec delay between the video and the audio (delayed).

To extract the original video into a audio and video composites look at the command on extracting audio and video from a movie

Here is more information of how to use ffmpeg:
http://www.ffmpeg.org/ffmpeg-doc.html

gist.vim: Vim Plugin For Gist

Gist is a simple way to share snippets and pastes with others. All gists are git repositories, so they are automatically versioned, forkable and usable as a git repository.

Yasuhiro Matsumoto’s Gist.vim plugin allow you to work with gist from Vim.

Working With gist.vim

:Gist

post whole text to gist.

Gist -p

post whole text to gist with private.

:Gist -a

post whole text to gist with anonymous.

:Gist -e

edit the gist. (shoud be work on gist buffer) you can update the gist with :w command on gist buffer.

:Gist -e foo.js

edit the gist with name ‘foo.js’. (shoud be work on gist buffer)

:Gist XXXXX

get gist XXXXX.

:Gist -c XXXXX.

get gist XXXXX and put to clipboard.

:Gist -l

list gists from mine.

:Gist -la

list gists from all.

Installing Gist.vim

1. Download the script
2. Move Gist.vim to ~/.vim (on Unix/Linux) or ~vimfiles (on Windows).
3. Restart Vim.

Shorten URLs using Python and bit.ly

Last time we found how to shorten URLs using Python and Google’s goo.gl URL shortening service. This time we will see how to use bit.ly’s api to shorten URLs. Here is the Python way of shortening/expanding URLs using using bit.ly. You will require a bit.ly user name and apikey to use this service. The apikey can be found here http://bit.ly/a/your_api_key. Also look at the complete bit.ly API Documentation.

#!/usr/bin/python
# use bit.ly's URL shortener
# requires urllib, urllib2, re, simplejson

try:
  from re import match
  from urllib2 import urlopen, Request, HTTPError
  from urllib import urlencode
  from simplejson import loads
except ImportError, e:
  raise Exception('Required module missing: %s' % e.args[0])

user = "username"
apikey  = "yourapikey"

def expand(url):
  try:
    params = urlencode({'shortUrl': url, 'login': user, 'apiKey': apikey, 'format': 'json'})
    req = Request("http://api.bit.ly/v3/expand?%s" % params)
    response = urlopen(req)
    j = loads(response.read())
    if j['status_code'] == 200:
      return j['data']['expand'][0]['long_url']
    raise Exception('%s'%j['status_txt'])
  except HTTPError, e:
    raise('HTTP Error%s'%e.read())

def shorten(url):
  try:
    params = urlencode({'longUrl': url, 'login': user, 'apiKey': apikey, 'format': 'json'})
    req = Request("http://api.bit.ly/v3/shorten?%s" % params)
    response = urlopen(req)
    j = loads(response.read())
    if j['status_code'] == 200:
      return j['data']['url']
    raise Exception('%s'%j['status_txt'])
  except HTTPError, e:
    raise('HTTP error%s'%e.read())

if __name__ == '__main__':
  from sys import argv
  if not match('http://',argv[1]):
    raise Exception('URL must start with "http://"')
  print shorten(argv[1])

Shorten URLs using goo.gl and Python

As we all know it, Google has its own URL shortening service called goo.gl. Google’s URL shortener still doesn’t have an official API and it doesn’t offer all the features that are available at bit.ly, but it works well.

Here is a Python script to shorten URL using goo.gl.

#!/usr/bin/python
# use Google's http://goo.gl/ URL shortener
# requires urllib, urllib2, re, simplejson
def shorten(url):
  try:
    from re import match
    from urllib2 import urlopen, Request, HTTPError
    from urllib import quote
    from simplejson import loads
  except ImportError, e:
    raise Exception('Required module missing: %s' % e.args[0])
  if not match('http://',url):
    raise Exception('URL must start with "http://"')
  try:
    urlopen(Request('http://goo.gl/api/url','url=%s'%quote(url),{'User-Agent':'toolbar'}))
  except HTTPError, e:
    j = loads(e.read())
    if 'short_url' not in j:
      try:
        from pprint import pformat
        j = pformat(j)
      except ImportError:
        j = j.__dict__
      raise Exception('Didn't get a correct-looking response. How's it look to you?nn%s'%j)
    return j['short_url']
  raise Exception('Unknown eror forming short URL.')

if __name__ == '__main__':
  from sys import argv
  print shorten(argv[1])

Usage:

$ python g.py http://segfault.in
http://goo.gl/Uh5h

Update:
Expand URLs

def expand(url):
  try:
    import urllib
  except ImportError, e:
    raise Exception('Required module missing: %s' % e.args[0])

  f = urllib.urlopen(url)
  return f.geturl()

How to set CPU affinity for a process in FreeBSD

Processor affinity means, on a multi-CPU machine, the process(es)run only on dedicated set of CPUs. In other words processes are bound to isolated (subset) of the CPUs. This feature can be usedduring performance benchmarking, and also while deploying an application.

To get the CPU model and number of active CPUs try the following command:

$ sysctl hw.model hw.ncpu

The cpuset command

The cpuset command can be used to assign processor sets to processes, run commands constrained to a given set or list of processors, and query information about processor binding, sets, and available processors in the system.

cpuset requires a target to modify or query. The target may be specified as a command, process id, thread id, a cpuset id, an irq or a jail id. Using -g the target’s set id or mask may be queried. Using -l or -s the target’s CPU mask or set id may be set. If no target is specified, cpuset operates on itself. Not all combinations of operations and targets are supported. For example, you may not set the id of an existing set or query and launch a command at the same time.

There are two sets applicable to each process and one private mask per thread. Every process in the system belongs to a cpuset. By default processes are started in set 1. The mask or id may be queried using -c. Each thread also has a private mask of CPUs it is allowed to run on that must be a subset of the assigned set. And finally, there is a root set, numbered 0, that is immutable. This last set is the list of all possible CPUs in the system and is queried using -r.

When running a command it may join a set specified with -s otherwise a new set is created. In addition, a mask for the command may be specified using -l. When used in conjunction with -c the mask modifies the sup- plied or created set rather than the private mask for the thread.

The options are as follows:

-c           The requested operation should reference the cpuset avail-
             able via the target specifier.

-g           Causes cpuset to print either a list of valid CPUs or, using
             -i, the id of the target.

-i           When used with the -g option print the id rather than the
             valid mask of the target.

-j jailid    Specifies a jail id as the target of the operation.

-l cpu-list  Specifies a list of CPUs to apply to a target.  Specifica-
             tion may include numbers seperated by '-' for ranges and
             commas separating individual numbers.

-p pid       Specifies a pid as the target of the operation.

-s setid     Specifies a set id as the target of the operation.

-r           The requested operation should reference the root set avail-
             able via the target specifier.

-t tid       Specifies a thread id as the target of the operation.

-x irq       Specifies an irq as the target of the operation.

Examples

Create a new group with CPUs 0-4 inclusive and run /bin/sh on it:

cpuset -c -l 0-4 /bin/sh

Query the mask of CPUs the is allowed to run on:

cpuset -g -p

Restrict /bin/sh to run on CPUs 0 and 2 while its group is still allowed
to run on CPUs 0-4:

cpuset -l 0,2 -p

Modify the cpuset /bin/sh belongs to restricting it to CPUs 0 and 2:

cpuset -l 0,2 -c -p

Modify the cpuset all threads are in by default to contain only the first
4 CPUs, leaving the rest idle:

cpuset -l 0-3 -s 1

Print the id of the cpuset /bin/sh is in:

cpuset -g -i -p

Move the pid into the specified cpuset setid so it may be managed with
other pids in that set:

cpuset -s  -p

Source: http://www.freebsd.org/cgi/man.cgi?query=cpuset