Web Submission and Review, Chair Documentation
Version 0.64 - June 2013
Table of Contents
- Installing the software
- Web-based installation
- Other installation methods
- First-time customization
- The administration page
- Managing the submission site
- Tweaking Server Settings
- Managing the review site
- Initial set-up
- Paper assignments and blocking access to papers
- Tracking progress
- Rebuttal phase
- Discussing a group of submissions
- Program-committee votes
- Wrapping up the PC work
- Managing the final-version submission site
- Adding invited talks to the program
1. Installing the software
One of the goals of this software was to make administration as easy as
possible, so you will not need to run scripts etc. In principle,
to install this software you just unpack the zip file in some directory
in your web tree (call it the BASE directory), point your browser at that
directory and off you go. However, this simple method of installation
requires the web-server to be able to write files in the web-tree,
which in some cases is considered a security risk. I therefore provide
also other installation methods, as explained below.
Prerequisites.
This is a PHP/MySQL software, which means that to use it you must have
a MySQL server and a PHP engine (and your web-server must supports PHP
and MySQL). It uses the php PDO extension to access the database, so
it requires version 5.0 or higher of php. Also, proper functioning of
this software depends on "proper" functioning of you web-server. In
particular, some issues that I came across are:
- The web-server (and the PHP engine) must be willing to upload files.
For example, in one system that I used to test it, the PHP engine was
configured to only allow uploads of files upto 500KB (and I was wrecking
my brains as to why I can only upload pdf and not postscript files..) In
my case it turned out that I had to change the parameter LimitRequestBody
in php.conf (which on that system was somewhere under /etc/httpd/).
Anoter parameter that I needed to update sometime is the limit
upload_max_filesize in php.ini, the default value for
this parameter is only 2MB, and some of the submission files can easily
be larger than this.
- The web-server must be able to send email. For example, if this is a
Windows machine then the variable SMTP in php.ini should be set correctly.
Some of the functionality of this package also depends on the availability
of external programs. Currently the only parts of the functionality that
need external programs are preparing an archive file with all the
submissions (which needs either the
Archive_Tar
PEAR library or the tar or zip programs), and stamping
the submissions with a header saying "do not ditribute" (which needs
either the Zend framework or auxilary program to convert between PDF
and Postscript). See some more details in the
implementation notes.
Experimenting with the software.
I wrote a small testing package for the purpose of experimenting with the
software. To use this package, copy the contents of the "testOnly"
subdirectory to the BASE directory where you installed the software itself
before you customize the installation. This testing package adds 53
dummy submissions and 17 dummy PC members to the conference database.
Troubleshooting.
Most of the parameters of the software are stored in the "parameters"
table in the database. The details of the database itself are stored in the
file init/confParams.php under the BASE directory. If you have
problems with the software, looking at these parameters may provide
some hints as to the source of the problem.
1a. Web-based installation
To use this method, simply copy the contents of the webtree directory
to somewhere in your web tree (call it the BASE directory), making sure
that the web-server has permission to write in the "init"
sub-directory of the BASE directory. After copying the files to the BASE
directory (and setting the permissions), point your browser to the page
chair/initialize.php under the BASE directory. In this form you
need to specify various details that are needed for initializing the new
site:
- Web Server.
The DNS name or IP address of the web server, as accessed by the users of
this software. This parameter is used when the software sends emails and
needs to specify the address where some pages can be found. (Note that
in some cases the administrator may use a different address. For example,
the administrator uses "http://localhost/confName/" while everybody
else uses "http://www.severName.org/confName/". In this case
the chair needs to enter the latter address on the customization page.)
- MySQL Server.
This parameter tells the PHP scripts where to look for the MySQL server. In
common installations the same host runs both the web-server and the MySQL
server. If this is not the case (i.e., the web server communicates with a
back-end database that is hosted on a different machine) then you need to
check the radio-button for "The MySQL server runs on a different host" and
provide the DNS name or IP address of the machine that hosts the MySQL server.
-
MySQL details.
There are three ways to create and populate a database for use with the
submission and review software:
-
- Create a new database.
If you have administrative access to the MySQL server, you can simply
enter the administrator username and password and the software will
automatically create a new database. The "administrator" whose details are
thus specified should have sufficient privileges to DROP and CREATE databases
and to GRANT access privileges on these databases.
- Create new tables in an existing database.
This is similar to the previous option, except it uses an existing
database instead of creating a new one. Several instances of the
software can share the same database, and they are distinguished by
different prefixes for the table names. For example, in one instance
we use the tables confXZYsubmissions,confXYZcommittee,...
while in the other we use ABC13submissions,ABC13committee,...
. To create new tables in an existing database, you need either
to specify the administrator username and password as above, or the
username and password of a user that have enough privileges to CREATE
tables in the specified database.
- Use existing database and tables.
Sometime you may prefer to create the database for this conference by
some other means. (For example, some hosted sites have a "control panel"
that lets users create new databases. In other sites one must ask the
administrator to create new databases.) In this case the software will
not attempt to create any table, but will assume that all the tables
were already created.
In the initialization page you need to provide either the administrator
username and password as above, or the username and password of a standard
user (where in the second case you can only use options 2 and 3 above,
and the user should have sufficient privileges for the case that you
choose). You also need to provide the name of the database to use (new
or existing), and the table-name prefix to use. If you provide the
username and password then a new MySQL user will be created, with
sufficient privileges to access (only) the tables of the current
instance.
-
UPLOAD directory. This is the local directory where the uploaded
submissions would be stored. If you do not specify anything in this field,
the UPLOAD directory defaults to the subdirectory "subs" under the
BASE directory. The UPLOAD directory need not be in the web-tree, but the web
server must be able to read and write in that directory.
- Program Chair emails.
In this field you can specify either just an email address or also the name
(in the form Name <email>). If you do not specify the chair's
name, it will default to something like "<conf-name> Chair".
The email addresses that you specify here will be used as the program chair's
username for the purpose of accessing the review site.
The chair's email will also be CC'ed on every email that is sent to authors
of submitted papers, including submission/revision confirmations, the final
decision notifications and the comments for authors. The system supports
multiple chair accounts, but during initialization you can only specify one
of them.
- Administrator email.
The administrator's email is rarely used by the software: it's main purpose
is to serve as a backup address for the chair's email. For example, if the
need arises to reset the chair's password, then an email with the new password
will be sent to both the chair and the administrator email addresses. Also,
there are some rare error situations where the software would sends an email
to the administrator. After it is set, the administrator email can only be
changed by manually editing the file init/confParams.php under the
BASE directory.
After Initialization
Once you hit the Submit button at the bottom of the initialization form,
you are taken directly to the customization page in order to customize the
site for the conference (see below). At this point
the web server no longer needs to write to the "init" subdirectory,
so you can change the permissions to make that directory read-only.
1b. Other installation Methods
One alternative method of installation (in Unix systems) is to change
directory to where you unzipped this software (above the "webtree"
and "tools" directories) and to issue the following command from
the command prompt:
% ./tools/mkNewConf.sh confname
where confname is a short-name for the conference (e.g., xyz2007).
It is recommended that you read the script "mkNewConf.sh" before
running it, and you can optionally modify it to set some default values.
This shell script will ask you for the locations of the BASE and UPLOAD
directories (and other parameters) and then it will install the software
in the BASE directory, setting the permissions as needed. The script will
then print out the username and password that are needed in order to access
the customization page. One limitation of this method is that it does not
support sharing a database between different conferences, rather it only
implements creating a new database for each new conference.
Another option is to copy the template parameter file which is provided
in subs/confParams.php into the init subdirectory,
then editing it, filling in all the relevant details, and only then
accessing the initialization web form as above. When invoked in this
setting, many of the fields in the initialization form are hidden (since
the relevant details are already specified in the parameter file), and
the software just uses the given parameter file instead of trying to
create it.
Yet another option is to just go through the web-based initialization
procedure as above without providing the parameter file. The procedure
will initialize the database and do everything else, but will fail to write
the parameter file to disk, and instead will output all the details to
the results web-page. You can then just cut and paste these parameters and
create the init/confParams.php file yourself, and then proceed with
customization as described next.
2. First-time customization
Before it can be used, the site still needs to be customized for the
conference. This is done from the page chair/customize.php under the
BASE directory. This form lets the program chair specify the conference
name, deadlines, program-committee members and many other parameters.
Links are provided from this form to this documentation page, where you can
find some additional explanations about the role of different parameters.
All the parameters that are specified in this form can be modified later
from the administration page. Some fields in the customization form that
require more explanation are described below:
- Pre-registration. Some conferences require
authors to register their submission before actually submitting it. Often the
deadline for registration is 1-2 weeks before the submission deadline. If you
leave the field for pre-registration deadline empty, then the software will
assume that pre-registration is not required.
- Opt-in text. Some conferences would like the option
for submissions to choose to opt in (via a checkbox) to some list or program.
For example, the text might read "This paper qualifies for the 'Outstanding
Young Author' award." Later in the process, the chairs will have the option
to list all papers who 'opted in' for further processing. If this is left
blank, the checkbox will be left out of the submission page.
- Supported Formats. The software is very
permissive in terms of handling formats other than the ones that you specify
here. If a submitter uploads a file in an unsupported format, the only
thing that happens is that the extension unsupported is added
to the file name. (For example, uploading a file xyz.txt, the
file will be stored as xyz.txt.unsupported.
- Reviewer Preferences. If you check this
box, PC members will see a link on the review page asking them to specify
their reviewing preferences. The chair will then be able to see all these
preferences and can take them into account in assigning submissions to
reviewers. Moreover, the chair will be given the option to let the software
automatically compute an assignment or submissions to reviewers based on
these preferences.
UNchecking this box will turn off reviewer preferences, and also disable
the automatic assignment option.
- Reviewer Attachments. Check this box to
allow the reviewers to upload a file as an attachment to their review. This
file is considered part of the comments to authors, and will be attached to
the comments that are emailed to the authors at the end of the review process.
Click here for some considerations for
and against allowing this option.
- See All Late Revisions. Check this box
to let reviewers see all the versions of submissions that were uploaded after
the deadline. The chair can always accept late updates to submissions, even
after the deadline. If this box is checked then the reviewers can see a list
of all the versions off of the submission page (i.e., what you get when
clicking on the submission title). If this checkbox is cleared then only
the latest version is available for the reviewers to see.
- External Communications. Checking
this box enables some features for communications with authors and external
reviewers. Specifically, PC members will see links on the discussion boards
for sending specific discussion items to the authors or to sub-reviewers,
where emails to authors are moderated by the chair(s) but emails to external
reviewers are not. The recipients get a link that allows them to respond
to the email, and their response is integrated directly into the discussion
board. Click here for more
information.
After Customization
Once you hit the Submit button at the bottom of the customization page,
you are taken to a confirmation page where you can once again verify all
the parameters. Then the software is customized with these parameters
and you are presented with a receipt page. That page includes the chair's
password and links to the submission page and the administration page. The
submission site is now ready to receive submissions, and you can use the
administration page to manage it.
3. The Administration Page
The administration page is what you use to manage the site and the entire
submission and review process. The links on that page change according
to the different phases of the process. You have one set of links during
the submission phase, another during the review phase, and yet others
during and after the final-version submission phase. You also have some
"general purpose" links there for changing your password, looking at the
error-log file, reading the documentation, etc.
You also have an "Undo/Redo" link that lets you either undo the last
change that you did from the administration page, or redo the last
change that you undid. For example, if you closed the submission site
and the decided to reopen it, you can use the Undo link to do that.
4. Managing the submission site
The administration page for the submission site is fairly simple. The
main things that you can do from that page are:
- You can see a list of all the (non-withdrawn) submissions.
- You can review and modify many of the submission-site parameters that
you specified during customization, including the conference submission
deadline.
- You can tweak some parameters of the web-site (mainly email settings)
as described below.
- If the conference requires pre-registration of submissions, then you
have a link to close the pre-registration. This prevents new submissions
from being registered, but does not close the submission site: Authors
who pre-registered earlier can still submit (and revise) their papers.
- Finally, when the submission deadline passes you can close the submission
site and activate the review site. Clicking this last link will take you to
a confirmation page, where you can confirm that you really want to close the
submissions. The same page also lets you modify some of the review parameters
that you specified during customization.
You will also see on the administration page links to some pages that
deal with review-period activities, such as setting the review guidelines,
managing PC membership and setting up PC votes. These are described
in more detail below.
4a. Tweaking Server Settings
This form is meant to provide some workarounds common bugs and
configuration errors (mostly related to email). It currently consists
of the following options:
Modify the server time. This provides a crude work-around in case the server′s internal clock is off. Here you can specify some "time shift" amount that will be added to the server's time.
Email header-line separation. RFC 2822 specifies that different
lines in the email header should be separated using the two characters
\r\n (CRLF). However, the documentation for the
PHP mail function
contains the following comment:
Note:If messages are not received, try using a LF (\n) only. Some poor
quality Unix mail transfer agents replace LF by CRLF automatically (which
leads to doubling CR if CRLF is used). This should be a last resort, as it
does not comply with RFC 2822.
Accordingly, the default setting that is used in the software is CRLF, but
you have the option to override that default.
X-Mailer Header Line.
Sending this header line is considered standard net etiquette, but some
spam-filters out there mark as spam anything that has this header line.
UNcheck this box if this causes email loss.
Sender Header Line.
If you know how the web-server is calling itself on email messages that
it sends, you can specify it here. This is often something like
userName@machineName where userName is the username of the
web-server (e.g., www, apache, admin, etc.) and
machineName is either the DNS name of the machine (e.g.,
www.mit.edu) or sometimes just localhost.
When the web-server sends email, the name of the web-server itself is set
as the "envelope sender" of this message. It was observed that some ISPs
filter as spam email messages where the From: header line does not
match the "envelope sender". Hopefully specifying a matching "Sender:"
line would prevent these emails from being discarded.
Specify "envelope sender".
Sometimes you can explicitly specify the "envelope sender" to be used, by
supplying the parameter "-f name-to-use" to the PHP mail function.
This only works on (some) Unix systems, and does not work when PHP is in "safe
mode". Also, using this option typically adds an "X-Warning:" header
line to the message, which can again increase the odds of this message being
discarded en-route. The PHP
documentation has the following comment about this method:
The user that the web server runs as should be added as a trusted user to
the sendmail configuration to prevent a 'X-Warning' header from being added
to the message when the envelope sender (-f) is set using this method. For
sendmail users, this file is /etc/mail/trusted-users.
5. Managing the review site
Once you close the submission site, you are taken back to the administration
page, where you can now administer the review site. At the top-right corner
you still have a "Submission-site" box, now consisting only of a few links
that let you list the submissions is various orders (and also revise or
withdraw submissions if need be). You can even submit new papers (just in
case you want to accept for consideration a submission that arrived after
the deadline.)
5a. Initial set-up
One of the first things that you would want to do after the deadline
(if not before) is letting the PC members access the review site. This
is done from the "Manage PC membership" page. On that page you can add
or remove PC members, reset the password of a PC member, modify the
email address or the name that is recorded in the database for each
member, and set the status of a member as either a chair or a regular
member. The PC members can login to the site using their email address
and password. When you reset the password of a PC member, an email
message is sent to that member with the new password. (Also, changing
the email address of a PC member automatically resets his/her
password.) Note that it is possible for several people to use the same
email address as long as they use different passwords (but all the
passwords will be sent to that one email address).
Other things that you can do from the administration page is to create
an archive file with all the submissions (so that reviewers do not need
to download each submission separately), and edit the review-guideline page.
The review guidelines page consists of the grade semantics, important dates
in the committee work (such as the individual review deadline), and other
review policies.
5b. Paper assignments and blocking access to papers
The software also lets you assign some submissions to each reviewer or
block reviewers from accessing specific submissions (e.g., blocking PC
members from accessing the reviews of their own papers).
If you did not disable that option, then PC members are given the option
to access a Reviewing-Preferences page where they can specify what
submissions they want or do not want to review, and they can also
specify if they feel that they have conflict of interest with some
submissions. All these preferences are made available to the chair,
should you want to use them in assigning or blocking access to submissions.
Note that access to a submission is NOT blocked automatically when the
reviewer specifies a conflict-of-interests. The chair must explicitly
block access from the Edit conflicts
page.
-
On the Edit conflicts page you
are presented with a list of all the PC members, and for each member
you are given a list of the submission-ID's that that member specified
as having conflict-of-interests with. For each PC member you can then
specify a list of submissions that this member should be barred from
seeing. You are then asked again to confirm these choices, and
once you do the submissions that you specified are removed from the view of
that PC member. (You can access the Blocking-Access page at any time during
the review process, should you need to modify these decisions.)
-
For the task of assigning submissions to reviewers, you are given the choice
to either assign everything manually, or to let the software compute an
assignment (using the reviewer's preferences) that you can later adjust.
Manually Assigning Submissions to Reviewers
There are two interfaces that you can use to manually specify the reviewing assignments. You can use either of these interfaces as you see fit, and once you "save the assignments" with one interface, the other interface is automatically updated as well.
- The matrix interface:
This interface consists of a matrix of check-boxes, with one column for each
PC member and one row for each submission. Throughout each column there are
running headers that lets you see the number of submissions that are assigned
to that PC member as well as a heuristic indication as to "how happy" this PC
member will be with this assignment. Also, at the right end of each row you
can see how many reviewers are assigned to that submission.
If you have Javascript enabled in your browser, then these sums and indicater
are updated every time that you check or clear a check-box, and the page trys
to asynchronously update the sketch assignments on the server with your changes
as you make them. This is a "best effort" approach, however, you will not
be notified of communication errors with the server. To ensure that the
assignment is recorded at the server you should use the "save assignments"
button at the bottom of the matrix, and this will also reload the page with
all the updated information. (To import changes made by another chair into
your matrix, just reload/refresh the page in your browser.)
The heuristic "indication of happiness" is computed as follows: The software computes the fraction of requested papers from all the assigned ones, and the faction of assigned papers from all the requested ones, and the "happiness level" is taken as the largest of these two fractions. A level below 50% is "Angry"
, 50-64% is "Sad"
, 65-79% is "Satisfied"
, and 80% and up is "Happy"
.
Every entry in the matrix also displays the preferences that the PC member specified for that submission, encoded as a number between 0 and 5 and colored from red to green (if nothing appears in the entry, it means that the PC member specified the default number 3). Also, if you specified some preferences of your own from the chair-preferences page, then the checkbox itself will be colored either red (should not review) or green (should review).
- The submission-list interface:
This interface consists of a list of submissions, where for each submission
you can specify a list of PC members that are assigned to that submission.
Automatically Computing Paper Assignments
If you did not disable that option, you will find a link off the main
assignment page that points to the auto-assignment page, where you can
have the software compute an assignment of submissions to reviewers.
The software uses an algorithm based on maximum-flow to generate this
assignment, as described next.
In addition to the reviewers specifying their preferences, the chair
can specify preferences as to who should (or shouldn't) review what
submission. Note that specifying that "I want Joe to review submission 53"
is different than manually assigning submission 53 to Joe. Specifically,
this is just a hint to the software and is not binding (which means that
you can specify as many preferences as you want without having to worry
about load balancing and such like). The chair preferences are used to
modify the reviewer preferences during the auto-assignment procedure as
follows:
-
When the chair indicates that a certain PC member SHOULD NOT review
a certain submission, the auto-assignment procedure will decrease that
preference. Denoting the preference-level that the reviewr assigned to
tha tsubmission by P (between 0 and 5), then the auto-assignment
procedure will treat it as max(0,P-2).
-
When the chair indicates that a certain PC member SHOULD NOT review
a certain submission, the auto-assignment procedure will increase that
preference. Denoting the preference-level that the reviewr assigned to
tha tsubmission by P (between 0 and 5), then the auto-assignment
procedure will increase it to min(6,P+1) if P was at least 3,
and will leave P unchanged otherwise.
Note that after the adjustments due to the chair preferences, the
auto-assignment algorithm treats the reviewer preferences as taken
from the range 0-6. Another adjustment that is made is that for
reviewers that do not have any submission at preference levels 5/6,
all their preference-4 submissions are elevated to preference-5. The
auto-assignment algorithm will not assign to a submission to a reviewer
if the (adujsted) preference level for it is less than 3.
After adjusting the preferences, the software prepares a flow network graph where each unit of flow from the source to the sink corresponds to an assignment of a submissions to a PC member. The graph has a source node, a sink node, and also four nodes for each reviewer (correpsonding to pref6 → pref5 → pref4 → pref3) and one node for each submission.
- There is a capacity-1 edge from prefP to every submission for
which the reviewer has (adujsted) preference level P.
- The capacity of the edges source → pref6 and pref6 → pref5
is the maximum load that can be added to this reviewer. That is, if each
PC member can be assigned upto 15 submissions, and you already assigned
two submissions to that PC member, then the capacity on these edges will
be set to 13.
- The capacity of the edge pref5 → pref4 is the load minus something, to force at least a few assignments with pref-5/6 for each reviewer. The "something" is set to half the smallest of the number of pref-5/6 submissions of this reviewer and the global load, rounded down.
- The capacity of the edge pref4 → pref3 is set to at least 20% of the load. However, it could be increased to make sure that the total of pref4 → pref3 capacity plus the number of pref-4/5/6 submissions is at least 120% of the total load for that reviewer.
- From each submission there is an edge to the sink with capacity
equal the number of reviewers that should be assigned to that submission,
minus the number of reviewers that are already assigned to it.
After setting this network, the assignment is computed in several steps.
First, a maximum flow is computed using only the pref-5/6 nodes, then
the pref-4 nodes are added and the flow is augmented, and finally the
pref-3 nodes are added and the flow is augmented again. The maximum
flow in each step is computed using the Edmonds-Karp/Dinic algorithm
[1],
which prefers short flow-paths over long ones. It is expected that
this process will assign submissions at higher preference levels when
possible, and "resort" to preference-level 3 only when there is no other
option.
Parameters of the algorithm.
Once you got the preferences from all the PC members and also specified
your own preferences, you are ready to let the software compute the initial
assignments for you. Before the algorithm is run, however, you have to
specify some parameters. In particular, you can specify a list of
excluded reviewers that will be hidden from the view of the algorithm.
This means that the algorithm will not assign any submissions to these
reviewers. However, submissions that you assigned to the manually before
running the algorithm will be kept, and will be counted against the
number of required reviews per submission. You can use this option,
for example, if one of the "PC members" on your list is the PC chair
(to whom you do not want to assign any submissions).
Other parameters that you need to specify are how many reviewers are
assigned for each submission and whether or not you want to keep existing
assignments (if any) or start from scratch.
Once you specify all the parameters, the software automatically computes
an assignment of submissions to reviewers, and you are taken to the
manual-assignment page where you can review and modify this assignment.
Sketch Assignments vs. Visible Assignments
The assignments that are automatically computed by the software or set
from the manual assignment page are stored as a "sketch copy" that is
only visible to the chair but not to the reviewers. This allows the
chair to play around with the assignments before making them publicly
available.
To allow the PC members to see their assigned submissions, the chair must
check the box to "Make these assignments visible to the reviewers" next to
the "Save Assignments" button on either the matrix or the list interface.
Hitting the "Save Assignments" button with this box checked will copy
the sketch assignments to the visible assignments, thus letting the
reviewers see their assigned papers.
Note: As opposed to paper assignments, setting or clearing
conflict-of-interests status for papers will immediately be visible to the
reviewers.
5c. Tracking progress
After you assigned the reviewing duties, you can track the progress of the
reviewing process off the administration page. The Overview page shows a
matrix of all PC members and submissions, and each entry in that matrix
shows whether a PC member is assigned to a submissions and whether it
entered a review of it. If the PC member entered a review for the submission,
you can click on the entry to see that review. At the bottom of the
page you can manage the "discuss flags" that let PC members see each
other's comments and participate in the discussions. You can also see each
PC members average scores across all the reviews they have given.
At any time you can go to the Set Submission Status page where you can
modify the status of any submission. The available status values are the
usual ones, namely "No status", "Reject", "Maybe Reject", "Discuss",
"Maybe accept", and "Accept". The Set Submission Status page has both a
"scratch" and "final" status. Only chairs can see the scratch status, and
it can be changed without affecting final status. Only when you click "Set
scratch and final status" will the final status be updated for all to see.
At the top of the Overview and Set-Status pages you can see how many
submissions are currently assigned any of the status values.
Also, at any time you can also go from the Administration page to the
Review page (via the links at the top and bottom of the page), where you
can participate in the review and discussions just like any other PC member.
5d. Rebuttal phase
At any point during the review phase you can choose to send comments-to-authors
that have been entered. In this email you are given the option to send a link to the submissions rebuttal page. Each submission is given one rebuttal with a maximum number of characters set by you. When the authors submit a rebuttal it is shown on the discussion page in below the discussions and above the reviews.
On the rebuttal-setup page you can choose how many characters it can have
and when is the deadline for submitting it, and you can open the rebuttal
(or close it, if it is currently open).
5e. Discussing a group of submissions
The chair can set-up discussion boards for the purpose of
discussing/comparing a group of submissions. These boards work just like any
other discussion board, except that reviewers that have a conflict with any
of the submissions in the group cannot access that board.
5f. Program-committee votes
The software includes support for setting up and managing program-committee
votes. You can have the committee vote on things such as "what papers to
move from 'Discuss' to 'Maybe Accept'", or "which of the accepted papers
should receive the best paper award", or even "who will give the invited
talk at the conference".
Managing the votes is done from the Voting page. On that page you can
set the parameters for a new ballot, edit the parameters of the ballots
that are currently in progress, and see a list of all the ballots (in
progress and otherwise) with a link to the result page for each one.
You can set up a ballot to vote either on submissions or on "other
things" (that you need to specify explicitly). Note that PC members can
only vote on submissions when they are in the public-discussion phase of
the review. Before that they can only vote on "other things" (e.g.,
invited speakers). Also, PC members will not see entries for submissions
that are blocked for them, even if these submissions are included in the
current vote.
When voting on submissions, you can either include all the submissions
in the vote, include only submissions from specific categories (e.g., only
submissions in the "Discuss" category), or explicitly list the submissions
that you want to include in the vote.
Note that if you choose to include in the vote, say, all the submissions
in the "Discuss" category, then the submissions that the PC members see
when they vote are the ones in the "Discuss" category at the time that
they are voting. Put in other words: adding or removing submissions
to the "Discuss" category in mid-vote will add or remove these submissions
from the ballot.
When you set up ballots, the PC members will see a link from their review
page that lets them participate in the current ballots, and you will be
able to track the voting results off the Voting page. You can also modify
the parameters of a vote, even when this vote is already in progress.
Note that if you modify the list of submission-IDs or the list of "other
things to vote on" in mid-vote, you must make sure that you do not modify the
order of items, since the software identifies vote-items with their position
in the list. For example, swapping the order of two items will result in each
of them being assigned the tally of the other.
When you close a vote, the tally becomes visible to the PC members via
a link off their review homepage. Note that only the tally is visible
to PC members, not the individual votes. Also, reviewers cannot see the
tally for submissions for which they have a conflict. You can prevent
even the tally from becoming visible by checking the box to "Hide tally
from PC" (next to the "Close vote" button) when you close the vote.
5g. Wrapping up the PC work
After all the decisions are made, the software lets you send to the authors
the notification of decisions and the comments-to-authors that were entered
during the review period. Just follow the appropriate link off the
administration page, and you will be given the opportunity to edit these
emails before they are sent. In addition to the contact authors, the chair
is also CCed on all these emails.
When sending the final comments-to-authors, you can check a box to let
the authors provide feedback on the reviews that they received.
Together with the reviews, the authors will get a link to page
where they can provide feedback on these reviews. The feedback on
each review (if any) is sent by email to the program chair(s) and the
PC-member who uploaded that review, and the program chair can also
see this feedback from the discussion page for that submission.
This can give you some picture (albeit partial) of the authors′
view of the reviews, for future references. For example is a reviewer
did a particularly good job --- or a particularly bad one --- you can
let future program chairs know about it.
Note that from the Notification page you can only send emails to
authors of submissions whose status is either "Accept" or "Reject".
The comments page (And the generic page for sending email to authors)
also give you the choice of sending emails to authors of submissions
with a particular status, or only to specific submissions that you
explicitly list. This can be used, for example, to hold off on sending
emails to some submissions by setting their status to something other
than "Accept" or "Reject", and later sending these submission separate
emails (e.g., for conditional-accept submissions).
If you want to keep all the reviews and discussions for future reference,
you can find a link off the administration page that packs all of them in
one text ascii page that you can then save for your files.
You also have a link to a page that lets you edit the final-version
preparation instructions (for authors of accepted papers), and a link
to close the review site and activate the site for final-version submission.
Once the review site is closed, PC members can still access the site and
read the comments and discussions, but they can no longer modify them.
6. Managing the final-version submission site
After you activated the final-version submission site, you still see a
"Review site" box at the top right corner of the administration page, that
lets you send emails to authors or change the status of submissions if
need be. Other than that, you can see a list of accepted papers (with
links to the final version when it is uploaded), create (and then download)
an archive file containing the final versions of all the accepted papers,
or send email to the authors of all the accepted papers (e.g., to remind
everyone that the deadline is in three days).
When the camera-ready deadline arrives, you can shutdown the final-version
submission site. After you do that, you are given yet another link that
automates the creation of the various indexes for the proceedings.
Specifically, you are given the option of specifying the order in which
the papers will appear in the proceedings and the length of each paper,
and then the software generates for you a LaTeX file (using Springer
llncs style) that includes the external-reviewer list, the list of
program committee members, the table-of-contents and the author index.
6.a Adding invited talks to the program
During the final-version submission phase you can add to the program invited
talks. Doing so will allow the invited speakers to upload their talk to
the server, to be included in the conference proceedings just like any
other accepted paper.
7. Feedback
I will very much appreciate any comments that you can give me regarding
this software. Please contact me at shaih@alum.mit.edu.