Early church fathers barely talked of papal infallibility and supremacy

The papacy was based on faith and the reading of scriptures.

Those are all serious historical arguments, and they reflect lines of reasoning that many historians and Orthodox theologians have made. Whether they ultimately succeed depends on how one interprets the evidence and the development of doctrine. Here's a breakdown:

1. Irenaeus and Augustine's silence on papal infallibility and supremacy
2. Your point is essentially an argument from historical silence. Both wrote extensively about the Church, authority, heresy, and doctrinal disputes.
3. Neither explicitly teaches supremacy or the doctrine defined at Vatican I: that the pope, under certain conditions, is preserved from error when defining doctrine for the whole Church.
4. Catholics generally respond that the doctrine was implicit and only later defined when controversy required precision.
5. Critics respond that if such a unique charism existed, one would expect at least occasional explicit discussion during major theological controversies.
6. Cyprian
7. Cyprian is one of the strongest pieces of evidence for those skeptical of modern papal claims.
8. During the rebaptism controversy he openly disagreed with Pope Stephen and insisted that each bishop possesses full episcopal authority.
9. He wrote that no bishop should set himself up as "bishop of bishops."
10. Catholics often point to Cyprian's high regard for the Roman See and his language about Peter, while critics emphasize that his actions seem inconsistent with belief in universal papal jurisdiction.
11. Communion with Rome
12. You're right that many early texts emphasize the importance of communion with Rome.
13. The difficult historical question is why.
14. Was Rome simply the most prestigious apostolic church?
15. Was it first among equals?
16. Was it a practical center of unity?
17. Or did it possess divinely instituted supreme jurisdiction?
18. The early sources often do not answer this question explicitly, leaving room for different interpretations.
19. Communion does not automatically imply supremacy
20. This is an important logical distinction.
21. Even if the early Church believed communion with Rome was normally necessary for catholic unity, that alone does not establish:
22. universal ordinary jurisdiction,
23. immediate jurisdiction over every bishop,
24. or papal infallibility.
25. Additional evidence would be needed to establish those stronger conclusions.
26. Binding and loosing
27. This is also an important exegetical point.
28. In Matthew 16, Peter uniquely receives the keys and the power to bind and loose.
29. In Matthew 18, Jesus explicitly gives the authority to bind and loose to the other apostles as well.
30. Catholics generally argue that the keys given uniquely to Peter signify a distinctive office in addition to the shared apostolic authority.
31. Orthodox and many Protestant interpreters argue that Matthew 18 shows the governing authority belongs to the apostolic college collectively, with Peter serving as a leading member rather than possessing supreme authority.

Overall, your argument highlights a cumulative case rather than relying on a single proof text:

1. The strongest papal doctrines are not clearly articulated in the earliest centuries.
2. Early bishops, including Cyprian, sometimes behaved as though they did not recognize universal papal jurisdiction.
3. Appeals to communion with Rome do not by themselves establish Vatican I's doctrines.
4. The New Testament distributes binding-and-loosing authority beyond Peter.

The church was successful. The Bible has stood the test of time and we have over 2 billion Christian’s followers.