Challenges: WL hierarchy has notatble limitations: coarse, qualitative, and may not well reflect practical requirements

They propose a novel framework for quantitatively studying the expressiveness of GNN architectures, addressing the above limitations.

Introduction

One of the most prominent drawbacks of popular GNNs lie in the limited expressive power. MPNN is bounded by the 1-dimensional WL in distinguishing non-isomorphic graphs.

— theory paper is hard to understand…

Human reviews are slow and of variable quality. Therefore, some people focus on using LLM to do reviewing.

Main Challenges:

1. risk of misuse

2. inflated review scores

3. overconfident ratings

4. skewed score distributions

5. limited prompt length

Their method:

1. without prompt engineering by using LLM watermarking to mark LLM-generated reviews
2. classifying and detection errors and shortcomings of papers
3. using long-context windows that include the review form, entire paper, reviewer guidelines, code of ethics and conduct, area chair guidelies, and previous year statistics
4. blind human evaluation of reviews.

Methods

1. Dataset: ICLR 2023 4956 papers with 18565 reviews.
2. Pipeline: During inference (solid lines), the authors upload a paper with the conference name. The paper is reviewed by OpenReviewer using GPT-4 with the conference statistics of the previous year, reviewer guidelines, code of ethics, code of conduct, full paper text, and review form. The LLM review is then watermarked and returned to the author, who provides feedback about the review.

Evaluation

1. randomly shuffle the papers and choose 10% of the papers randomly
2. randomly permute each of the reviews for each paper, which consists of three human reviews and three LLM reviews
3. blindly evaluate the reviews by human experts. The experts do not know whether the review being evaluated was written by human or an LLM, and each expert answers four meta-questions about each review.

Prompt Samples

Summary: Summarize what the paper claims to contribute. Be positive and constructive.
Correctness: Please classify the paper on the following scale to indicate the correctness of the technical claims, experimental and research methodology and on whether the central claims of the paper are adequately supported with evidence. 4: All of the claims and statements are well-supported and correct. 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct. 2: Several of the paper’s claims are incorrect or not well-supported. 1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.
Strenths and weaknesses: List strong and weak points of the paper. Be as comprehensive as possible.

Clarity, Quality, Novelty, and Reproducibil- ity: Please evaluate the clarity, quality, novelty, and reproducibility of the paper.

Summary of the review: Clearly state your initial recommendation (accept or reject) with one or two key rea- sons for this choice. Provide supporting arguments for your recommendation.

Some of my thoughts about reviewGPT

Prompt: From the review, list the strength and weaknesses from the following points: clarity, quality, novelty, and reproducibility

Prompt: From the last year’s all reviews, cluster the reivews and scores, have a model, then use this model to predict this year’s one paper submission, what is the review, and what is the score, and whether it would be accepted.

Definition of AI agents: artificial entities that sense their environment, make decisions, and take actions.

A general framework that can be tailored for different applications. Three scenarios: single-agent, multi-agent, and human-agent cooperation.

Exploring the behavior and personality of LLM-based agents.

Background

A general conceptual framework for the LLM-based agents with three key parts: brain, perception, and action. Brain is composed of a large language model. Perception serves a similar role that of sensory organs for humans. Action that agents can better respond to environmental changes and provide feedback, and even alter and shape the environment.

Trends in Agent Research

Symbolic agents: An example is knowledge-based expert systems.

Reactive agents: reactive agents do not use complex symbolic reasoning. They primarily focus on the interaction between the agent and its environment, emphasizing quick and real-time responses.

Reinforcement learning-based agents: e.g. Alphago. Agents can autonomously learn in unknown environments, without explicit human intervention.

Why LLM is suitable as the primary component of an agent’s brain?

autonomy: an agent operates without intervention from humans or others and possesses a degree of control over its actions and internal states.

reactivity: an agent can refer to its ability to respond rapidly to immediate changes and stimuli in its environment.

Interactive engagement between human and agent

human-agent interaction involves agents collaborating with humans to accomplish tasks.

Evaluation for LLM-based agents

Utility: primary criterion. Success rate of task completion.

Scaling up the number of agents

However, current research predominantly involves a limited number of agents, and very few efforts have been made to scale up the number of agents to create more complex systems or simulate larger societies.

Generative agents can simulate believable human behavior. These agents remember and reflect days past as they plan for next days.

This architecture extends LLM to store the complete record of the agents’ experiences and retrieve them dynamically to plan behavior.

observation, planning, and reflection: three parts, each contribute critically to the believability of agent behavior.

Introduction:

memory stream – a long-term memory module that records, in natural language, a comprehensive list of the agent’s experiences

reflection – synthesizes memories into higher- level inferences over time, enabling the agent to draw conclusions about itself and others to better guide its behavior.

planning – translates those conclusions and the current en- vironment into high-level action plans and then recursively into detailed behaviors for action and reaction

We manifest a small society of twenty five agents in a game environment. End users can observe and interact with these agents.

e.g. One agent wants to hold a valentine’s party. Then the system does not need to tell each agent to come to this party but this agent will automatically spread this information.

Method:

1. instantiate them as characters in a simple sandbox world reminiscent of The Sims
2. For each role, write a background introduction for it.
3. Agents communicate with each other in full natural language. Agents are aware of other agents in their local area, and the generative agent architecture determines whether they walk by or engage in conversation.
4. Users and agents can influence the state of the objects in this world, much like in sandbox games such as The Sims. For example, a bed can be occupied when an agent is sleeping, and a refrigerator can be empty when an agent uses up the ingredients to make breakfast

New discoveries – Emergent Social Behaviors

1. Information diffusion: Sam tells another agent he wants to chase for mayor
2. reflection: after reflection, architecture can make more believable decisions
3. future directions: create more powerful simulations of human behavior to test and prototype social systems and theories, and to create new interactive experiences

1. Abstract
   1. Task: predict properties about new consequent events based on similar cause-efect events present in the KG
   2. Methodology: Frame the task to a 2-hop LP task, where the first hop is a causal relation connecting a cause event to a new effect event and the second hop is a property about the new event which we wish to predict
   3. Dataset: The model is applied on a novel dataset containing newsworthy events with causal relations curated from Wikidata, where EvCBR outperforms baselines in- cluding translational-distance-based, GNN-based, and rule-based LP models.
2. Intro
   1. Motivation: A major limitation of most LP methods is that relations can only be predicted among entities which already exist in KG
   2. Key idea: leverage examples of past events in the KG to perform LP for properties of unseen entities.
   3. Methodology: We frame our problem as a 2-hop link prediction task – e.g., starting from a cause event, we assume the existence of a causal relation to a new efect event and predict its properties
   4. Contribution summary
      1. leveraging knowledge about similar cause effect events to make predictions about the unseen effect
      2. curate and release a novel dataset surrounding causal events in Wikidata, extracting news events that are con- nected by causal relations as well as their local connections.
      3. our model shows superior performance on our event dataset as well as com- petitive performance on a modifed evaluation dataset based on the FB15k-237 dataset.
3. Experiments
   1. Problem formulation
      1. Predict missing tail which is not in the knowledge graph
   2. Case retrieval
      1. Entity similarity: For each entity, formulates a vector contains the number of entities dimensions

1. Abstract
   1. Task: predict entities that appear less frequently in knowledge graphs.
   2. Contribution 1: propose knowledge relational attention network to leverage the graph context by simultaneously projecting neighboring triples to diferent latent spaces and jointly aggregating messages with the attention mechanism
   3. Contribution 2: propose knowledge contrastive loss by combining the contrastive loss with cross entropy loss, which introduces more negative samples and thus enriches the feedback to sparse entities
2. Intro
   1. Motivation: predicting entities rarely appear in knowledge graphs is still challenging. They investigate in-degree and link prediction performance (Maybe in-degree is a point I can investigate more on)
3. Methodology
4. Experiments
   1. sparse knowledge graphs performance
   2. Entity In-degree Analysis
   3. Combination of Different GNN Encoder and Projection Head
5. Conclusion
   1. They present KRACL model to alleviate the widespread sparsity problem for knowledge graph completion
   2. introduce knowledge contrastive loss to introduce more neg- ative samples, hence more feedback is provided to sparse entities

1. Before feature engineering, use clean_tweet function to takes a raw tweet as input and removes URLs, user mentions, special characters, and extra spaces. The output is a cleaned tweet with only the relevant textual content.
2. Then, use tokenizing function of pre-trained models to normalize the tweets.
3. Then, get embeddings of each tweet using pre-trained models.

Bert Model Choices: Bert, Bertweet, XLNet

1. load the pre-trained BERTweet model tokenizer and model of bertweet-base
2. Read clean_text column and use tokenizer to get tokens of each cleaned tweet
3. Convert tokens to a tensor and generate bert_tweet embeddings
4. For each tweet, use average of all 768-dimension embedding of tokens to get a tweet embedding: 768-dimension
5. Store the first 100 tweets’ results to 100_tweets_with_bertweet_embeddings.csv

1. Do not ask it to extract contents from a website, it is not connected to the internet. The content it returns is from its current dataset until 2021.
2. If you want to ask it to help me write python code, it is better tell it which module you want it to import. The code quality will improve a lot.
3. Before asking it, think more about what do I want. Wrong prompt results in wrong answer.

中间debug花了一点时间，遇到的问题是create a new file but no permission. 解决方法是用 chmod 777命令更改文件夹的权限。命令在下面

sudo chmod -R 777 /home/sixven/camp_sms/inputs

20/3/2023 更新

刚开始的时候还在问google能不能跑在aws s3上面，其实这个根本就不对。s3只是个存储。今天（一个礼拜之后）才想起来去问问chatgpt，算来浪费了一个小时。要拥抱gpt! 今后chatgpt要变成我的default option了。

然后是我给的这个 https://pawelurbanek.com/ethereum-node-aws 有问题的，因为这个没有加debug mode.后面再调整也不太可的样子。所以今天重新开了一个ec2 instance.

教训是需要去看官方文档，goethereum.org的东西，当时有点懒因为这个网站被nus给ban了所以没去上面double check. 偷一会儿懒后面就要用更多的时间去补了。