1= Introduction

1.1 Purpose of the System

The purpose of computer science portal is to let any= one in need of any kind of computer science related information can search for the desired information without encountering any unrelated data from the World = Wide Web. As research aspect, the purpose of the research is to have more exact = and better extracted data after web crawling process and to offer fast and efficient results to the user.

1.2&n= bsp;Design Goals =

Computer Science Portal Project’s main design = goals can be classified as performance, reliability and efficiency.

1.2.1 Performance:

· Response Time: The portal must be qui= ck to respond and the results of a search should be returned very fast. This w= ill be provided using invert indexed database.

· Sea= rch Performance: Because of the methods used like R-FC, user gets more accu= rate and more relevant results in less time that he/she will get from ordinary search engines currently operational in the net.

· Data Gathering: The time of process of creating database is feasible because unlike in the ordinary search engines, R-FC helps system to visit only cs related web pages throughout the interne= t.

· Hardware Adequacy: The server that provides the portal access and creates the background database should be a = fast and reliable computer in order to fulfill the performance requirements.

1.2.2 = Dependability:

· Reliability: The system should be reliable so that gives working web directories as result to the users with eliminating old and out of action links.

· Availability: The system should be available 24 hours a day to serve its users continuously.

1.2.2<= span style=3D'mso-bookmark:_Toc123718293'> = Maintenance

· Ext= ensibility: It should be easy to add some new feature to the system such as additional search interests and upgrades.

· Mod= ifiability: The system should be built so that with certain changes of the system requi= res very smaller amount work to do.

· Ada= ptability: The application should be adaptable to different application domains such t= hat building another portal of different search focused on another science topi= c.

· Rea= dability: The system should be readable in terms of understanding the semantics of the codes.

1.3 = Defin= itions, acronyms and abbreviations

FC: Focus Crawling

R-FC: Rule= -based Focus Crawling

IE: Information Extraction

IR: Information Retrieval

1.4 Refer= ences

1) S.Chakrabarti, “Mining the Web Discovering Knowledge from Hypertext Data.” Mor= gan Kaufman Publishers, 352 pages, 2003.

2) Can, F., Altingovde, I. S., Demir, E., Effici= ency and effectiveness of query processing in cluster-based retrieval, Information Systems.

3) I. S. Altingovde, Ö. Ulusoy, Exploiting Interclass Rules for Focused Crawling, IEEE Intelligent Systems, vol.19, no.6, 2004.

4) Peter Flach and Nicolas Lachiche. Naive Bayesian classificati= on of structured data., Machine Learning, Vol. 57, Number 3, pages 233-269, 2004

5) Fuhr, N. Models for retrieval with probabilistic indexing. Information Processing= and Management 25, 1 (1989), 55--72.

6) http://www.nist.gov/

7) http://www.cs.cmu.edu= /~mccallum/bow/rainbow/

8)http://www.cs.utexas.edu/users/hyukcho/= classificationAlgorithm.html#Classification%20Algorithms

9) ht= tp://www.cs.bris.ac.uk/Publications/pub_info.jsp?id=3D2000266

1.5 Overview

The search portals currently operational on World Wi= de Web helps people to find what they look for and search portals are the most vis= ited sites among all other. They help internet users but it is likely that they = will give hundreds of unrelated pages as results in addition to a couple of actu= ally desired links. Because their base is the entire web and they search the wri= tten word in this huge area. It is most likely that the first result with the highest relevancy that you get for CS will be a gaming site about the pc ga= me Counter Strike. So, our target is creating a web server that will be a guid= e to all cs-related people from professional to academic scale that overcomes irrelevant result problem by carefully extracting only CS related sites from the World Wide Web and by giving users results found around these CS sites.=

Also required research to create backbone of this sy= stem will be based on rule-based focused crawling approach with an extension in = the “building the document collection phase”, information retrieval method namely “inverted-index-based search” for the basic-keyword search, information extraction methods for the advanced search.

2 Current Software Architecture=

Our project consists of three main parts. These are crawling part, information extraction part and making advanced queries on downloaded pages. There are some project and applications that are working = on these subjects. However, most of them deal with only one part of this proje= ct like just crawling or information extraction.

Websphinx is tool to crawl the web that is developed= in Carnegie Mellon University. Websphinx only offers normal crawling option, not domain specific crawling.= It is does not have politeness option which aims not to straiten the web serve= r. But, Websphinx has user friendly interface and it is customizable by user.

However, there are some projects that have similar a= ims and similar design issues with our project, which is called CrossMarc devel= oped in Demokritus, Greece National Center for Scientific Research. Ruled-based focus crawler implementation, coded by Omer Rauf Atay, is software that we mostly benefit from.

2.1 CrossMarc

CrossMarc project consists of four main parts. These= are; 1-Collection of domain specific web pages 2- Information extraction from do= main specific web pages 3- Data storage, storing information extracted from page= s to database. 4- To present data to end user through a multilingual user interf= ace. Although, we aim to information extraction for only English, CrossMarc has facility for English, Greek, Italian and French. CrossMarc uses focus-crawl= ing in order to gather domain specific web pages. Cross-lingual name matching techni= ques are used while performing information extraction. Demonstration of CrossMarc can be seen in http:/= /www.iit.demokritos.gr/skel/crossmarc/demo-images.htm. However, CrossMarc does not have IR with inverted index which is first part= of second phase of our project.

2.2 Ruled-Based Focus Crawler

Ruled-base focus crawler is crawler that has normal crawling, focus-crawling and ruled-based focus crawling options. We are goi= ng to use this crawling code for first phase of our project. We are planning to add new features to crawling code as a research issue. This crawling softwa= re does not have sub-document scoring features. Details of extended version of R-FC by sub-document scoring can be seen in Appendix A.

3 Proposed Software Architecture

3.1 Overview

The Computer Science Portal that we propose will be a reliable, accurate, effective and efficient system. Users will get ranked r= esults to their queries in remarkably acceptable response times. Since only the relevant parts of the web will be crawled and since only the relevant docum= ents will be indexed, keyword based search utility of our system will be effecti= ve. In the same way, the information extraction part will be done in coherently classified documents and so the portal will be very effective for structured queries. The portal will handle lots of preprocessing on the downloaded raw data before the querying phases: an inverted index file will be prepared for keyword based search, and classification & information extraction tools will be utilized to prepare structured information for dbms use for sophisticated querying. So both the keyword based querying and the sophisticated querying phases will be efficient with the use of inverted indexing and dbms.

3.2. Subsystems and System Decomposition of CSRP =

3.2.1. Subsystems of CSRP

► = PHASE1: Document Collecting:

In this stage of the project the aim is to reach only the domain relevant regions of WWW and to download all the domain relevant pages. Rule based focused crawling is utilized to accomplish this task. The crawling begins with some seed pages and continues by assigning relevance values to the links of subparts of a page with the use of Rainbow Library for classification. So some links in a page can be followed and some links of a page can be neglected. This property is an extension that will m= ake the crawler more focused and will make the downloaded web-page set more coherent. A second library used in this stage is the BerkeleyPQ. Since every page is assigned a relevance value according to its similarity with CS doma= in, all the links to be followed are inserted into the priority queue and the m= ost relevant ones are followed first. So the documents are collected in this wa= y, which finishes the phase 1.

► = PHASE2: Processing the Document Collection:

In this stage the downloaded pages in the= first stage are to be processed. The processing will in two ways: information retrieval as the first one and information extraction as the second one.

► = PHASE2.1: Information Retrieval:

In this stage the document vector is proc= essed and an inverted index file is created. The keyword based search will use th= is inverted index file that will make the querying efficient. Since the number= of keywords in a keyword based search is limited, the use of inverted index fi= le becomes quite efficient.

► = PHASE2.2: Information Extraction:

1-Classi= fier: Before applying information extraction on web pages, we first categorize the documents. After the defining the rules of classification apply classificat= ion. Naïve Bayesian classification algorithm will be used. Documents are separated into “personal home page”, “course page”, “and research group page”, “department home page”. Rainbow library is used as a classifier, details of rain bow classifier was mentioned in the analysis report. As an input, classifier accepts web page, after categorization of web pages according extracted rules, output of this module is document list for each classes.

Diagram 1) Input-Output Relationship of the Three Phases of the = CSRP

2- Informat= ion Extractor: After grouping of web pages, information extraction algorith= m is applied on these pages. Information extraction algorithm we are going to use relies on “message understanding”. Performing message understan= ding on free-format task is a quite difficult task, but by using Hidden Markov M= odel (HMM), one of IE techniques, we are planning to extract necessary data from pages. HMM is a finite state machine with probabilistic state transition. A= gain as input information extractor accepts web pages that are in the same categ= ory. Information extractor will be trained for each class differently. For examp= le, extracting “name” from “personal home page” will be performed according to different rules than rules that are used for extract= ing “name” from “course page”. Output of information extractor is “Extracted data vector” which will be directly sto= red into related table in the database. “Extracted data vector” contains data with their accuracy scores and document ID so that we will be able evaluate query result according to these scores.

► = PHASE3: Querying the Portal:

In this stage the querying operations are executed. There are two kinds of querying options in the Computer Science Resource Portal. The first one is the web-search-engine-like keyword-based querying, and the second one is a more sophisticated querying composed of SQL-like query commands.

► = PHASE3.1: Keyword Based Querying:

Web-search engine like querying is provid= ed in the phase. The user will enter some keywords to search for. The system will take the query and execute it on the inverted index file, returning the most relevant documents with the ranking order. The inverted index file sustains= the portal with the required efficiency in the response time since the number of words that are searched for are averaged to a small number.

► = PHASE3.2: Sophisticated Querying:

In this phase a user will specify the que= ry on the query interface and enter some limitations. Then the system will convert this choice into commands that the dbms can understand. So the software in phase 3.2 is the middle tier between the user, who deals only with the clie= nt interface, and the dbms that receives its commands from our software. Then = the dbms will return the results to our software which will prepare the result = page to be shown to the client.

3.2.2 System Decomposition

The subsystem decomposition of the Computer Science Resource Portal is differing with the three stages of the whole project. In= the first phase there is only one main frame that executes the crawling job. Th= is computer possesses a huge memory and disc as crawling process requires thes= e. The second phase of the project works on the document collection that phase= 1 produced. This phase requires the parsing and indexing of all the documents= and very few user interactions are required, so again a mainframe is used for t= his phase. The memory and disc requirement for this mainframe is also the same.= The machine in the third phase is to control the client – server interact= ion, so is the middle tier between the clients and the dbms. This machine doesn&= #8217;t need to have as large memory or disc capacity as the mainframes in the first two phases. Deployment Diagram for Phase 3 can be seen in Diagram 2.

Diagram 2: UML Deployment Diagram<= /a> of <= /span>Subsystem Decomposition of the CSRP

= &nb= sp; This Diagram shows the 3^rd phase requirements of the port= al. The first and the second phases require mainframes with Linux OS to do the required calculation.

3.3 Hardware/Software Mapping

For different phases of the project hardware software requirement will be different.

Crawling part of the project requires more than 512 = MB memory and 2 GHz processor. Internet connection is also another requirement for crawling, since pages are downloaded from internet.

Second phase does not demand internet connection; be= cause it will be performed on already downloaded documents. Phase two also does n= ot have interactive relation with end user, speed of system is not first matte= r. However, creation of inverted index requires at least 1 GB memory, as there will be large dataset. Storing, retrieving and interacting with data will be handled by My SQL server.

Phase three is interactive part and it should offer = fast service time to end user. So we need at least 2 GHz processor for server in order to be able to meet requirement. Also, since we offer online search, internet connection is necessity for system. Open source software, Apache w= eb server will be used since end users will be able to use search portal on internet.=

For all three part of the project, compilation of C = codes will be done by gcc complier in Linux operating system.

3.4 Data Management

The system crawls the web, retrieves raw = data and then extracts desirable information into tables. So we use two types of databases.

1. First type is used while crawling, storing crawl rules, seen URLs, their contents, resolved DNSs - host addresses and priority queues. These databases are used for three purposes.

a. = Checking new URLs against already downloaded ones to prevent duplications

b. Storing resolved DNS and Host addresses so the crawler will not have to resolve the same addresses again and again.

c. = Priority queues are stored into databases so crawler can have resume feature by reconstructing the in memory queues using these databases.

2. Second type is used for storing the extracted information into tables. These tables are used by the actual search engine to provide answers to queries generate= d by users.

&n= bsp;

Unless a refresh condition is specified (= i.e remove pages which are older then ‘a’ days from ‘seenR= 17; database which will enable re-downloading that page), the data stored in the tables will be static. Users of CS portal will interact with the informatio= n by only performing SELECT type queries using the User Interface provided as web pages.

For the first type of databases we are us= ing BerkeleyDBs since they are going to be generated and used by the application program itself as the crawling process continues.

Figure 1: Berkeley Databases in the Crawling = Phase

For the second type of databases we will = use MySql. Tables created in MySql can be queried with using SQL and they will = be more efficient for search queries generated by the web pages.

3.5 Access control and security

&nb= sp;

Research-test-debug part of the project i= s run on a server in Bilkent. Team members and supervisors have the necessary acc= ess privileges to develop or modify the source code. Backups of the whole proje= ct are created at different times so we maintain older versions in case we nee= d to roll back. The server is not open to public so security is not an issue.

Once the web portal goes online the situa= tion will be different. Anyone with an internet browser will be able to access t= he portal. There is no user-password restriction since this is a search portal= and user specific information is not required perform any queries. Since the queries that can be performed on the data tables are pre-defined by the developers, users have no way of damaging or modifying the data using the u= ser interface provided.

The data tables will be created and modified by the = system admin. The admin will have necessary privileges to perform operations on databases using his/her username-password so it is very unlikely that anyone will be able to compromise the security.

3.6 Global software control

&nb= sp;

The crawler part of the system has a well defined flow. Until a predefined objective is fulfilled the program crawls = the web and saves data. Examples of these objectives can be to reach a maximum number of pages to crawl, time or crawling depth.

Information extraction again uses the same logic. Once it is run, the program will try to find and extract information= of predefined interests from crawled and downloaded data. Then data tables wil= l be created. Again some values can be used to constraint the time the program w= ill run.

The web portal part of the system is event driven. Once a user visits our page first the default page will be displaye= d. Then the system will wait for a user event. Types of these events are quite simple. User will mainly click links and buttons (i.e search) which in turn will cause the system to display a new page according to this new request. = If a search button is clicked, system will dynamically create a new page and dis= play it.

3.7 Boundary conditions

The web portal will run on a apache web s= erver. The application is initiated when a user requests a web page and terminates when the user closes the connection (i.e closing the web browser). Since no user specific data is used a crash will not affect any data.

The application part, which is the crawle= r and data extractor, goes through a simple flow. Program starts and runs until a predefined condition is met. If an immature termination occurs the crawler = can use the ‘priority queue’ databases to reconstruct the memory structures and resume crawling.

4 Glossary

● Rul= e Based Focused Crawling: This a type of web crawler that is an extension of focused crawling approach with tunneling. Instead of deciding whether to fo= llow or neglect the links on a page at the first level, with the use of rules th= is crawler check two level relevance values. The multiplication of the relevan= ce values for these two levels is the main concern in rule based focused crawl= ing that is why it can support tunneling. This type of crawling is used in our system.

● Doc= ument Vector: A document vector is the file containing the information= of which terms occur how many times in which documents. The document vector is converted into inverted file index to be used efficiently in keyword based querying. Since the number of terms in a keyword based query is limited, us= ing inverted file index is much more efficient than using the document vector during the querying phase.

● Inv= erted Indexing: An inverted index is an index structure storing a mapp= ing from words to their locations in a document or a set of documents, giving f= ull text search. An inverted index is the most important data structure used in search engines.

There are two main types of inverted indexes: An inv= erted file index contains for each word a list of references to all the documents= in which it occurs. A full inverted index additionally contains information ab= out where in the documents the words appear. The one used in out project is the inverted file index, containing the information of which terms occur in whi= ch documents.

= ;

5 Appendix

In focused crawling approach, each page is assigned a relevance value with the domain and if that value is lower than a threshold value the links in that page is not followed. However assigning o= nly one value to a very big web page is sometimes not a good idea, since differ= ent subparts of the page can be different than the whole characteristics of the page. In our implementation this approach is taken into consideration and e= ach page is divided into structural subparts that are supposed to be coherent in context. Each subpart is assigned a relevance value, and the links in these subparts are followed or not according to the relevance value of the corresponding subpart that the link belongs to. Having done so, the focused crawler used in the 1^st phase of the system will become even more focused than the focused crawler of in the literature, and so this will be = our contribution to the computer science world.

Some of the examples of the mentioned sub grouping approach can be seen in F= igure A.1 & Figure A.2.

Fig= ure A.1: Subpart Division of a research page of a university<= span style=3D'font-size:12.0pt;font-style:normal;mso-bidi-font-style:italic'>

This example has 13 subparts. 12 of them are shown with red-rectangles and the o= ther 1 subpart is the rest of the whole page without these 12 subparts.

Figure A.2: Subpart Division of the homepage of = an academician

This example has 6 subparts. 5 of them are shown with red-rectangles and the oth= er 1 subpart is the rest of the whole page without these 5 subparts.