Research in most disciplines involves studying natural or sociological phenomena in order to understand them better. Information Technology research is concerned with developing new theories and mechanisms to improve current practice in constructing computer systems. In this respect IT is similar to engineering: while some IT research involves only ideas or theory, this is rare. It is far more often the case that ideas are validated through practical implementation and evaluation. It should not be assumed, however, that all IT research focuses on computing systems rather than on human factors. In applied or action research, projects address specific problems in the real world that can be improved through creative use of computers; much research is devoted to improving the human interface to computing; etc. In these projects, research methods used in commerce and the humanities become part of the repertoire of IT researchers as well.
Most research projects comprise four main stages. These do not occur strictly in sequence. There is considerable iteration involved, as insights gained in later stages enable earlier decisions to be revised or reconsidered. The four main stages are:
identify a research question to address
collect information on the problem and on existing knowledge relevant to it
explore and evaluate one or more possible answers
report the results and conclusions
Not all of these steps will apply to all projects. Some of the best research simply identifies a new problem, without investigating answers. No research project can omit the last stage however; if work is not published it cannot be part of the body of knowledge in the discipline.
Should you embark on a research project? What can you expect from the experience? Some of the main attractions of research are the wonder and excitement invoked by building, observing or understanding something that no-one has created, seen or understood before. Research success is a major personal triumph that leads to recognition, and the privilege of adding to human knowledge. Researchers enjoy the thrills of discovery, the ability to meet and work with others who are deep thinkers and passionate about their subject, including (if not locally, then at conferences and seminars) invaluable talks with leaders in the field who will pass on their enthusiasm and their insight. In research, you have an enormous degree of freedom in choosing what you want to work on, who you want to work with, how you will do so and how you will manage your time. Moreover, project outputs will bring about change and improvement in the lives of others. While research is mainly about innovative ideas and experimentation, most of a researchers time is spent doing things that indirectly support and contribute to this: reading, planning, talking, conducting meetings, writing, etc. Researchers have to deal with frustrations and disappointments as well, and often the cause is beyond the researchers control (failure or inaccessibility of underlying technology/tools, inadequacy or privacy of expected data, lack of resources, etc.). At other times, experiments fail, hypotheses turn out to be wrong, or ones results are considered incorrect, insignificant or a duplication of others.
Research should not be conducted in isolation. Where projects are tackled by a single individual who does not meet regularly with any others, the person has the extra burden of putting effort into obtaining feedback and discussing new developments in the field. Working in a team provides a network for support and shared responsibility; it enables larger projects to be tackled and makes it easier for one to specialize in specific aspects because there are others to address remaining issues; and it develops teamwork skills, which are vital both within and outside the research sphere. The downside is that the overall quality and rate of progress are highly sensitive to the performance of others, and group management overhead is introduced. Managing a research team requires that individual roles and responsibilities are made clear to all, authorship of articles fairly agreed upon from the outset and reviewed when necessary, and regular meetings held with appropriate records/minutes.
All research should be seen as part of a much larger quest to better understand and harness human knowledge and experience in a discipline. Each project is part of a broader process - building on what has gone before, and adding ideas, theories and artifacts that can in turn be built on by others. In IT, this development typically comprises the following phases:
A new problem, new constraints, new opportunity or new approach is identified
Initial solutions, methods, algorithms, designs, theorems, programs, architectures, hardware or models are proposed
These are evaluated and refined, and different improvements to many of these ideas are investigated
Solutions are compared
A framework or taxonomy of the problem and solution space is devised; theorems are proved about the limits on any solutions; the existence of an optimal solution is considered and compared with current solutions; etc.
The best solutions are adopted by the community, commerce, industry and researchers in other disciplines or fields
Not all of these phases occur in all research some problems are too expensive to evaluate, or too large and varied to draw any conclusions about the solution space.
IT research results or products take the following forms:
a literature review
programs
architectures
systems or prototypes
theorems
models
user/expert surveys
user/expert experiments
system content (ontologies, knowledge bases, class libraries, graphics toolboxes, etc.)
measurements
hardware
analysis of existing research in the area
published papers and books (without this, the research remains incomplete!)
Research can be
Pure or applied
Qualitative and/or quantitative
Empirical (experimental) and/or analytical
Laboratory or action research
An applied research question has a clear and definite benefit in pointing to a solution or improvement for a practical problem. In contrast, when the solution to a research problem has no evident bearing on a real-life situation, but is of interest to the research community to give them better understanding, this is termed pure research. The cost of not knowing the answer to a pure research question is hard to measure as it revolves around the cost of lacking knowledge, which affects our ability to solve other, and more significant, research problems. But the cost of having no answer to an applied research problem can be measured in terms of money, health, lifestyle or the like. Whereas pure research addresses something we wish to know (or know better), applied research tackles something we wish to do (or do better). A researcher should be aware of whether they are interested in pure or applied research, and should not try to mix the two. For example, researchers should not relate their work to a practical problem if it is pure research and is not, in fact, demonstrably necessary in order to solve that practical problem.
Quantitative research is concerned with analysis done in terms of numbers; qualitative data describes artifacts and events scientifically without numbers. Qualitative research is more exploratory in nature, involving procedures such as interviews and observation. Quantitative research measures or counts pre-determined properties and phenomena; its results are easier to interpret.
Empirical methods are based on observation (surveys, case studies, experiments). These observations should be accurate and as general as possible (i.e. applicable to as many cases as possible). Such research can formulate a new theory or test an existing theory to confirm or refute it. Analytical methods involve modeling, simulation, formal proofs and analysis; they permit phenomena to be studied without building prototype systems or involving human participants.
Action research studies a social situation with a view to improving it; for example research directed at improving education or health care. It is problem-focused and context-specific, and involves a cycle of research, change and evaluation. Information gathering centres on understanding situations as ongoing processes happening in the real-world environment. Problem definition and analysis is an important aspect of action research; it aims to give a clear picture of the possible benefit of research on a practical aspect of our lives, and then studies how this might be achieved.
Laboratory research involves controlled measurement and interpretation of data in order to test hypotheses and theories. The problem with such controlled experiments is that an artificial environment is different from reality. The benefit of the scientific method of laboratory experimentation is that careful control enables accurate, reliable and repeatable studies to be done; with extraneous factors kept constant and independent variables systematically altered by the researcher.
The main research methods that can be employed to answer a research question are listed below. More than one of these methods will be used in most research projects, in order to tackle the same problem/task from a number of different perspectives:
literature reviews
prototyping
modelling
laboratory experiments and tests
field studies (observation and experimentation in real environments)
case studies (detailed analysis of a specific application)
surveys (interviews and questionnaires to assess human factors in IT)
mathematical proofs
heuristic evaluation
usability studies
statistical analysis
Each of the eleven methods listed above is a skill to learn, rather than a recipe to follow. There are difficulties to overcome, choices to make, and time/resource constraints requiring simplifying assumptions. Subsequent chapters will address some of these issues.
Technical research goals are most often met through the use of models, languages, algorithms and mathematical proofs; goals involving the human side of computing are typically met using surveys, case studies and experiments.
Note that programming does not constitute research, but software prototypes play an important role in experimental IT research. They enable properties to be studied, ideas to be validated, alternatives to be compared, measurements to be made, usability analysis to be conducted and field/case studies to be done. They are also a valuable source of new problems and insight.
Study papers in your subject area to see the methods used to conduct and to present research; it is best to use the same methods if you aim to publish in that field. The chances of choosing the right methods are greater, since they have proved successful for others; it will also be easier to convince the community of your research results if the methods used are what they expected.
As an example of how different research methods can be employed, consider the following situation. The research question: “How can the new technology <T> be adapted to run on very small computers?” can utilize the various methods in the following ways:
a literature review will familiarize you with <T>, with the properties of small computers, and with any work already done on using <T> with small machines
a prototype implementation of <T> on a small device can demonstrate its feasibility, evaluate and compare alternative implementations, and/or indicate what problems need to be solved in this new context
a model of the new technology or of small computers can be built and studied to enable current and future research to understand more about the problem
a prototype implementation can be tested and performance measurements recorded and analysed
the use of small computers in a particular real life context can be studied to determine the viability and demands on <T> in this domain, or a prototype can be used in the field and problems and limitations observed
a specific use of <T> on a particular device in some real world application can be studied to determine problems and potential solutions/improvements
an assessment of <T> by users with small machines such as mobile computers can be determined by surveying a representative sample etc.
The particular methods chosen will depend on how much or how closely existing research has already studied <T> in the context of small machines. It will also be governed by the interests of the researcher (e.g. some prefer to focus on low-level system implementation issues, others who prefer the human side may look at usability aspects, etc.), the availability of expertise and resources, the skills the researcher has and the skills s/he wishes to learn, etc. The only methods that are always essential are conducting a literature review at the outset, and publishing results on completion of the work. These first and last skills are imperative for anyone entering the research field.